[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:56:02 dist-test-slave-h4g5 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:56:02 dist-test-slave-h4g5 krb5kdc[399](info): set up 2 sockets
Jun 19 08:56:02 dist-test-slave-h4g5 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 08:56:03.860976 392 mini_kdc.cc:121] Time spent starting KDC: real 2.300s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260619 08:56:04.002164 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.141s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260619 08:56:04.318797 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.317s user 0.001s sys 0.000s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260619 08:56:04.618825 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.300s user 0.001s sys 0.000s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/krb5kdc/test-user.keytab.
W20260619 08:56:05.031705 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.413s user 0.001s sys 0.000s
Jun 19 08:56:05 dist-test-slave-h4g5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:56:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:56:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 08:56:05.075910 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42037
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:05.155369 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:05.155589 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:05.155619 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:05.156821 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:56:05.156864 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:05.156881 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:05.156898 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:56:05.156912 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:56:05.158398 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:05.158643 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:05.158870 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:05.164338 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:05.164382 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:05.164340 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:05.164364 412 server_base.cc:1034] running on GCE node
I20260619 08:56:05.164772 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:05.165066 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:05.166210 412 hybrid_clock.cc:648] HybridClock initialized: now 1781859365166191 us; error 48 us; skew 500 ppm
Jun 19 08:56:05 dist-test-slave-h4g5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:05.171192 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 08:56:05.171685 412 webserver.cc:460] Webserver started at http://127.0.98.62:37565/ using document root <none> and password file <none>
I20260619 08:56:05.171938 412 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:05.171984 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:05.172151 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:05.173254 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "656cd205704142d6918c16a10f93f698"
format_stamp: "Formatted at 2026-06-19 08:56:05 on dist-test-slave-h4g5"
server_key: "219301bfd626d882ead5d0b011a7f338"
server_key_iv: "bbe8111dfcade589f28eb98e1d22195b"
server_key_version: "encryptionkey@0"
I20260619 08:56:05.173604 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "656cd205704142d6918c16a10f93f698"
format_stamp: "Formatted at 2026-06-19 08:56:05 on dist-test-slave-h4g5"
server_key: "219301bfd626d882ead5d0b011a7f338"
server_key_iv: "bbe8111dfcade589f28eb98e1d22195b"
server_key_version: "encryptionkey@0"
I20260619 08:56:05.174976 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:56:05.175662 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:05.175844 412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:05.175933 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "656cd205704142d6918c16a10f93f698"
format_stamp: "Formatted at 2026-06-19 08:56:05 on dist-test-slave-h4g5"
server_key: "219301bfd626d882ead5d0b011a7f338"
server_key_iv: "bbe8111dfcade589f28eb98e1d22195b"
server_key_version: "encryptionkey@0"
I20260619 08:56:05.175997 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:05.186015 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:05.188192 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:05.188344 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:05.192075 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42037
I20260619 08:56:05.192142 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42037 every 8 connection(s)
I20260619 08:56:05.192485 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:56:05.193337 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:05.195585 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 656cd205704142d6918c16a10f93f698: Bootstrap starting.
I20260619 08:56:05.196162 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 656cd205704142d6918c16a10f93f698: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:05.196393 480 log.cc:824] T 00000000000000000000000000000000 P 656cd205704142d6918c16a10f93f698: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:05.197026 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 656cd205704142d6918c16a10f93f698: No bootstrap required, opened a new log
I20260619 08:56:05.198555 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 656cd205704142d6918c16a10f93f698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656cd205704142d6918c16a10f93f698" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42037 } }
I20260619 08:56:05.198693 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 656cd205704142d6918c16a10f93f698 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:05.198721 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 656cd205704142d6918c16a10f93f698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 656cd205704142d6918c16a10f93f698, State: Initialized, Role: FOLLOWER
I20260619 08:56:05.198832 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 656cd205704142d6918c16a10f93f698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656cd205704142d6918c16a10f93f698" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42037 } }
I20260619 08:56:05.198906 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 656cd205704142d6918c16a10f93f698 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:56:05.198932 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 656cd205704142d6918c16a10f93f698 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:56:05.198966 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 656cd205704142d6918c16a10f93f698 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:05.199452 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 656cd205704142d6918c16a10f93f698 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656cd205704142d6918c16a10f93f698" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42037 } }
I20260619 08:56:05.199542 480 leader_election.cc:302] T 00000000000000000000000000000000 P 656cd205704142d6918c16a10f93f698 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 656cd205704142d6918c16a10f93f698; no voters:
I20260619 08:56:05.199663 480 leader_election.cc:288] T 00000000000000000000000000000000 P 656cd205704142d6918c16a10f93f698 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:56:05.199714 483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 656cd205704142d6918c16a10f93f698 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:05.199875 483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 656cd205704142d6918c16a10f93f698 [term 1 LEADER]: Becoming Leader. State: Replica: 656cd205704142d6918c16a10f93f698, State: Running, Role: LEADER
I20260619 08:56:05.199895 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 656cd205704142d6918c16a10f93f698 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:56:05.199997 483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 656cd205704142d6918c16a10f93f698 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656cd205704142d6918c16a10f93f698" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42037 } }
I20260619 08:56:05.200399 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 656cd205704142d6918c16a10f93f698 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "656cd205704142d6918c16a10f93f698" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656cd205704142d6918c16a10f93f698" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42037 } } }
I20260619 08:56:05.200476 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 656cd205704142d6918c16a10f93f698 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:05.200536 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 656cd205704142d6918c16a10f93f698 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 656cd205704142d6918c16a10f93f698. Latest consensus state: current_term: 1 leader_uuid: "656cd205704142d6918c16a10f93f698" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "656cd205704142d6918c16a10f93f698" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42037 } } }
I20260619 08:56:05.200613 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 656cd205704142d6918c16a10f93f698 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:05.200687 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 412
I20260619 08:56:05.200785 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 08:56:05.200882 492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:56:05.201387 492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:56:05.201589 392 external_mini_cluster.cc:1428] Setting key 0bb92b95fc0cf2a8c0fffa9a3b8dd912
I20260619 08:56:05.203110 492 catalog_manager.cc:1347] Generated new cluster ID: ce3cfeaa021341bbb97b25566ed7e2ef
I20260619 08:56:05.203162 492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 08:56:05 dist-test-slave-h4g5 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:05.215204 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:05.202345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52096 (local address 127.0.98.62:42037)
0619 08:56:05.202512 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:05.202521 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:05.202805 (+ 284us) server_negotiation.cc:407] Connection header received
0619 08:56:05.204645 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:05.204652 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:05.204696 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:05.204808 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:05.206242 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:05.207192 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:05.208212 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:05.208619 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:05.208716 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:05.211881 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:05.211886 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:05.211886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:05.211897 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:05.213453 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:05.213840 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:05.213843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:05.213843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:05.213930 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:05.214165 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:05.214166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:05.214167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:05.214407 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:05.214565 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:05.214809 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 08:56:05.214946 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":76,"threads_started":1}
I20260619 08:56:05.215615 492 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:56:05.216347 492 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:56:05.221016 492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 656cd205704142d6918c16a10f93f698: Generated new TSK 0
I20260619 08:56:05.221201 492 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 08:56:05.251312 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42037
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:05.368619 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:05.368911 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:05.368965 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:05.370821 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:05.370896 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:05.370944 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 08:56:05.373208 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42037
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:05.373507 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:05.373772 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:05.379170 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:05.379186 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:05.379228 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:05.379348 506 server_base.cc:1034] running on GCE node
I20260619 08:56:05.379576 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:05.379854 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:05.381029 506 hybrid_clock.cc:648] HybridClock initialized: now 1781859365381013 us; error 28 us; skew 500 ppm
Jun 19 08:56:05 dist-test-slave-h4g5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:05.383615 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 08:56:05.384037 506 webserver.cc:460] Webserver started at http://127.0.98.1:40705/ using document root <none> and password file <none>
I20260619 08:56:05.384258 506 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:05.384311 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:05.384408 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:05.385426 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-0/data/instance:
uuid: "32a04116ab0c4c259b505dd0a27872f0"
format_stamp: "Formatted at 2026-06-19 08:56:05 on dist-test-slave-h4g5"
server_key: "47c626c03af72ea2f34792a8f532d0cc"
server_key_iv: "1078280ff66dfd81ebd1d67812ec3cd6"
server_key_version: "encryptionkey@0"
I20260619 08:56:05.385725 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-0/wal/instance:
uuid: "32a04116ab0c4c259b505dd0a27872f0"
format_stamp: "Formatted at 2026-06-19 08:56:05 on dist-test-slave-h4g5"
server_key: "47c626c03af72ea2f34792a8f532d0cc"
server_key_iv: "1078280ff66dfd81ebd1d67812ec3cd6"
server_key_version: "encryptionkey@0"
I20260619 08:56:05.386994 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:56:05.387763 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:05.388027 506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:05.388139 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-0/wal
uuid: "32a04116ab0c4c259b505dd0a27872f0"
format_stamp: "Formatted at 2026-06-19 08:56:05 on dist-test-slave-h4g5"
server_key: "47c626c03af72ea2f34792a8f532d0cc"
server_key_iv: "1078280ff66dfd81ebd1d67812ec3cd6"
server_key_version: "encryptionkey@0"
I20260619 08:56:05.388206 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:05.404775 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:05.406909 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:05.407024 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:05.407210 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:05.407474 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:05.407511 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:05.407532 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:05.407548 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:05.412820 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37007
I20260619 08:56:05.412891 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37007 every 8 connection(s)
I20260619 08:56:05.413182 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
I20260619 08:56:05.417131 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 506
I20260619 08:56:05.417222 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-0/wal/instance
I20260619 08:56:05.417379 392 external_mini_cluster.cc:1428] Setting key 6dec0cea10dd0488d96db882df18fae6
Jun 19 08:56:05 dist-test-slave-h4g5 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:05.422572 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:05.413920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57723 (local address 127.0.98.62:42037)
0619 08:56:05.413984 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:05.413986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:05.414159 (+ 173us) server_negotiation.cc:407] Connection header received
0619 08:56:05.414626 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:05.414628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:05.414640 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:05.414691 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:05.415618 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:05.416137 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:05.417255 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:05.417541 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:05.417609 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:05.419917 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:05.419918 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:05.419918 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:05.419925 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:05.421216 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:05.421662 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:05.421662 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:05.421663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:05.421708 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:05.421954 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:05.421955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:05.421955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:05.422065 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:05.422110 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:05.422429 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 08:56:05.422502 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 08:56:05.422854 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:05.413875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42037 (local address 127.0.98.1:57723)
0619 08:56:05.414040 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:05.414078 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:05.414550 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:05.414746 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:05.414749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:05.415342 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:05.415549 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:05.415554 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:05.416226 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:05.416227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:05.417171 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:05.417173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:05.417619 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:05.417620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:05.417752 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:05.418255 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:05.418264 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:05.419823 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:05.421290 (+ 1467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:05.421298 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:05.421301 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:05.421594 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:05.421746 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:05.421747 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:05.421747 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:05.421897 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:05.422117 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:05.422124 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:05.422298 (+ 174us) client_negotiation.cc:769] Sending connection context
0619 08:56:05.422423 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 08:56:05.422599 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260619 08:56:05.423391 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42037
I20260619 08:56:05.423496 635 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:05.423692 635 heartbeater.cc:511] Master 127.0.98.62:42037 requested a full tablet report, sending...
I20260619 08:56:05.424263 444 ts_manager.cc:194] Registered new tserver with Master: 32a04116ab0c4c259b505dd0a27872f0 (127.0.98.1:37007)
I20260619 08:56:05.424881 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57723
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 08:56:05.442680 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42037
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:05.521953 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:05.522162 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:05.522185 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:05.523294 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:05.523330 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:05.523355 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 08:56:05.524752 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42037
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:05.524935 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:05.525112 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:05.529871 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:05.529917 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:05.530067 640 server_base.cc:1034] running on GCE node
W20260619 08:56:05.529943 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:05.530321 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:05.530558 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:05.531720 640 hybrid_clock.cc:648] HybridClock initialized: now 1781859365531688 us; error 48 us; skew 500 ppm
Jun 19 08:56:05 dist-test-slave-h4g5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:05.534191 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 08:56:05.534633 640 webserver.cc:460] Webserver started at http://127.0.98.2:34161/ using document root <none> and password file <none>
I20260619 08:56:05.534826 640 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:05.534871 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:05.534983 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:05.536021 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-1/data/instance:
uuid: "4760f3c47c8e4989ae34489a73490e8e"
format_stamp: "Formatted at 2026-06-19 08:56:05 on dist-test-slave-h4g5"
server_key: "23e2f84039683462055e329a31e5b638"
server_key_iv: "e563e5b8f9dedc846076053b2b1fee44"
server_key_version: "encryptionkey@0"
I20260619 08:56:05.536406 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4760f3c47c8e4989ae34489a73490e8e"
format_stamp: "Formatted at 2026-06-19 08:56:05 on dist-test-slave-h4g5"
server_key: "23e2f84039683462055e329a31e5b638"
server_key_iv: "e563e5b8f9dedc846076053b2b1fee44"
server_key_version: "encryptionkey@0"
I20260619 08:56:05.537767 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:05.538655 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:05.538897 640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:05.539008 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-1/wal
uuid: "4760f3c47c8e4989ae34489a73490e8e"
format_stamp: "Formatted at 2026-06-19 08:56:05 on dist-test-slave-h4g5"
server_key: "23e2f84039683462055e329a31e5b638"
server_key_iv: "e563e5b8f9dedc846076053b2b1fee44"
server_key_version: "encryptionkey@0"
I20260619 08:56:05.539083 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:05.556389 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:05.558173 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:05.558295 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:05.558521 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:05.558815 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:05.558848 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:05.558876 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:05.558894 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:05.564867 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38067
I20260619 08:56:05.564929 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38067 every 8 connection(s)
I20260619 08:56:05.565222 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
I20260619 08:56:05.567415 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 640
I20260619 08:56:05.567514 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-1/wal/instance
I20260619 08:56:05.567675 392 external_mini_cluster.cc:1428] Setting key 09c8d26a13421e482f7418b01bcf9c12
Jun 19 08:56:05 dist-test-slave-h4g5 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 08:56:05.575980 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:05.565912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42547 (local address 127.0.98.62:42037)
0619 08:56:05.565966 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:05.565968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:05.566204 (+ 236us) server_negotiation.cc:407] Connection header received
0619 08:56:05.566676 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:05.566677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:05.566688 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:05.566737 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:05.567795 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:05.568506 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:05.569829 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:05.570168 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:05.570242 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:05.572846 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:05.572849 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:05.572849 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:05.572857 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:05.574443 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:05.574933 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:05.574934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:05.574934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:05.574993 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:05.575285 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:05.575286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:05.575286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:05.575404 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:05.575455 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:05.575830 (+ 375us) server_negotiation.cc:299] Negotiation successful
0619 08:56:05.575912 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 08:56:05.576231 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:05.565867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42037 (local address 127.0.98.2:42547)
0619 08:56:05.566073 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:05.566115 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:05.566598 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:05.566794 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:05.566797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:05.567385 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:05.567670 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:05.567677 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:05.568589 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:05.568591 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:05.569742 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:05.569744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:05.570267 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:05.570268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:05.570386 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:05.570954 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:05.570975 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:05.572723 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:05.574530 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:05.574539 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:05.574541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:05.574836 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:05.575066 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:05.575067 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:05.575067 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:05.575212 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:05.575485 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:05.575492 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:05.575668 (+ 176us) client_negotiation.cc:769] Sending connection context
0619 08:56:05.575790 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 08:56:05.575978 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":117,"threads_started":1}
I20260619 08:56:05.576678 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42037
I20260619 08:56:05.576781 769 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:05.576944 769 heartbeater.cc:511] Master 127.0.98.62:42037 requested a full tablet report, sending...
I20260619 08:56:05.577292 444 ts_manager.cc:194] Registered new tserver with Master: 4760f3c47c8e4989ae34489a73490e8e (127.0.98.2:38067)
I20260619 08:56:05.577672 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42547
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 08:56:05.594492 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42037
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:05.674142 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:05.674355 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:05.674378 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:05.675515 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:05.675558 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:05.675585 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 08:56:05.677032 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42037
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:05.677251 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:05.677469 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:05.682262 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:05.682257 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:05.682279 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:05.682503 774 server_base.cc:1034] running on GCE node
I20260619 08:56:05.682708 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:05.682909 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:05.684037 774 hybrid_clock.cc:648] HybridClock initialized: now 1781859365684026 us; error 29 us; skew 500 ppm
Jun 19 08:56:05 dist-test-slave-h4g5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:05.686359 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 08:56:05.686779 774 webserver.cc:460] Webserver started at http://127.0.98.3:36639/ using document root <none> and password file <none>
I20260619 08:56:05.686969 774 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:05.687023 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:05.687144 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:05.688190 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-2/data/instance:
uuid: "7359594208e8448fbcee65be8a0803c0"
format_stamp: "Formatted at 2026-06-19 08:56:05 on dist-test-slave-h4g5"
server_key: "1d1c4f5584bd897ef109bd37c381c13b"
server_key_iv: "6e8579bd11ede9e0723cfdf624e21754"
server_key_version: "encryptionkey@0"
I20260619 08:56:05.688513 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7359594208e8448fbcee65be8a0803c0"
format_stamp: "Formatted at 2026-06-19 08:56:05 on dist-test-slave-h4g5"
server_key: "1d1c4f5584bd897ef109bd37c381c13b"
server_key_iv: "6e8579bd11ede9e0723cfdf624e21754"
server_key_version: "encryptionkey@0"
I20260619 08:56:05.689711 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:05.690371 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:05.690522 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:05.690600 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-2/wal
uuid: "7359594208e8448fbcee65be8a0803c0"
format_stamp: "Formatted at 2026-06-19 08:56:05 on dist-test-slave-h4g5"
server_key: "1d1c4f5584bd897ef109bd37c381c13b"
server_key_iv: "6e8579bd11ede9e0723cfdf624e21754"
server_key_version: "encryptionkey@0"
I20260619 08:56:05.690659 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:05.705268 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:05.707355 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:05.707492 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:05.707698 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:05.707999 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:05.708032 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:05.708060 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:05.708101 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:05.713927 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34553
I20260619 08:56:05.713977 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34553 every 8 connection(s)
I20260619 08:56:05.714272 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
I20260619 08:56:05.718943 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 774
I20260619 08:56:05.719017 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859361540516-392-0/minicluster-data/ts-2/wal/instance
I20260619 08:56:05.719161 392 external_mini_cluster.cc:1428] Setting key 3736657fae97a354db23971de9abeb11
Jun 19 08:56:05 dist-test-slave-h4g5 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:05.723361 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:05.715317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56281 (local address 127.0.98.62:42037)
0619 08:56:05.715384 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:05.715386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:05.715569 (+ 183us) server_negotiation.cc:407] Connection header received
0619 08:56:05.716018 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:05.716019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:05.716029 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:05.716104 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:05.716967 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:05.717453 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:05.718367 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:05.718551 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:05.718594 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:05.720924 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:05.720926 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:05.720926 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:05.720934 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:05.722197 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:05.722571 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:05.722572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:05.722572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:05.722620 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:05.722804 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:05.722804 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:05.722804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:05.722898 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:05.722945 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:05.723235 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 08:56:05.723293 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 08:56:05.723582 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:05.715281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42037 (local address 127.0.98.3:56281)
0619 08:56:05.715446 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:05.715487 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:05.715942 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:05.716168 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:05.716172 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:05.716709 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:05.716899 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:05.716904 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:05.717524 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:05.717525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:05.718307 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:05.718309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:05.718607 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:05.718608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:05.718707 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:05.719236 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:05.719247 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:05.720789 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:05.722274 (+ 1485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:05.722280 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:05.722281 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:05.722512 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:05.722665 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:05.722666 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:05.722666 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:05.722761 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:05.722972 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:05.722978 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:05.723125 (+ 147us) client_negotiation.cc:769] Sending connection context
0619 08:56:05.723218 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 08:56:05.723345 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
I20260619 08:56:05.724000 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42037
I20260619 08:56:05.724107 903 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:05.724265 903 heartbeater.cc:511] Master 127.0.98.62:42037 requested a full tablet report, sending...
I20260619 08:56:05.724596 444 ts_manager.cc:194] Registered new tserver with Master: 7359594208e8448fbcee65be8a0803c0 (127.0.98.3:34553)
I20260619 08:56:05.724993 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56281
I20260619 08:56:05.731323 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 08:56:05 dist-test-slave-h4g5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859365, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 08:56:05 dist-test-slave-h4g5 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859365, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 08:56:05.752681 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:05.745338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44952 (local address 127.0.98.1:37007)
0619 08:56:05.745532 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:05.745536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:05.745548 (+ 12us) server_negotiation.cc:407] Connection header received
0619 08:56:05.745576 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:05.745577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:05.745591 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:05.745672 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:05.746214 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:05.747009 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:05.747708 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:05.748065 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:05.748153 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:05.750190 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:05.750199 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:05.750199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:05.750205 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:05.751551 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:05.751893 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:05.751896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:05.751897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:05.751953 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:05.752143 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:05.752144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:05.752145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:05.752291 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:05.752423 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:05.752494 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 08:56:05.752569 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":90,"threads_started":1}
W20260619 08:56:05.753033 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44952
Jun 19 08:56:05 dist-test-slave-h4g5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 08:56:05 dist-test-slave-h4g5 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 08:56:05.773712 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:05.767114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44968 (local address 127.0.98.1:37007)
0619 08:56:05.767176 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:05.767177 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:05.767241 (+ 64us) server_negotiation.cc:407] Connection header received
0619 08:56:05.767292 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:05.767293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:05.767306 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:05.767359 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:05.767870 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:05.768431 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:05.769135 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:05.769358 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:05.769417 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:05.771588 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:05.771589 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:05.771590 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:05.771596 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:05.772819 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:05.773122 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:05.773123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:05.773123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:05.773170 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:05.773363 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:05.773363 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:05.773363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:05.773445 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:05.773536 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:05.773586 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 08:56:05.773644 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 08:56:05.774166 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 506
I20260619 08:56:05.778182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 640
I20260619 08:56:05.783026 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 774
I20260619 08:56:05.786803 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 412
2026-06-19T08:56:05Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4251 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:56:05 dist-test-slave-h4g5 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:56:05 dist-test-slave-h4g5 krb5kdc[917](info): set up 2 sockets
Jun 19 08:56:05 dist-test-slave-h4g5 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260619 08:56:06.735432 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 08:56:06 dist-test-slave-h4g5 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:56:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:56:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 08:56:06.846892 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37761
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37761
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:06.925412 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:06.925647 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:06.925679 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:06.926919 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:56:06.926968 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:06.926988 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:06.927004 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:56:06.927018 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:56:06.928607 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:06.928869 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:06.929090 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:06.933856 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:06.934037 930 server_base.cc:1034] running on GCE node
W20260619 08:56:06.933856 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:06.933871 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:06.934368 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:06.934653 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:06.935808 930 hybrid_clock.cc:648] HybridClock initialized: now 1781859366935792 us; error 28 us; skew 500 ppm
Jun 19 08:56:06 dist-test-slave-h4g5 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:06.938369 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 08:56:06.938809 930 webserver.cc:460] Webserver started at http://127.0.98.62:41273/ using document root <none> and password file <none>
I20260619 08:56:06.939030 930 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:06.939074 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:06.939203 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:06.940316 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "e04e1b71380645a8ae77963a0ebb0f5d"
format_stamp: "Formatted at 2026-06-19 08:56:06 on dist-test-slave-h4g5"
server_key: "8f46f2401bf4044da246540ec900f7b3"
server_key_iv: "43b3b14fa4a8808d1d4ab9dac206cb20"
server_key_version: "encryptionkey@0"
I20260619 08:56:06.940653 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "e04e1b71380645a8ae77963a0ebb0f5d"
format_stamp: "Formatted at 2026-06-19 08:56:06 on dist-test-slave-h4g5"
server_key: "8f46f2401bf4044da246540ec900f7b3"
server_key_iv: "43b3b14fa4a8808d1d4ab9dac206cb20"
server_key_version: "encryptionkey@0"
I20260619 08:56:06.941967 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:56:06.942710 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:06.942857 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:06.942937 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "e04e1b71380645a8ae77963a0ebb0f5d"
format_stamp: "Formatted at 2026-06-19 08:56:06 on dist-test-slave-h4g5"
server_key: "8f46f2401bf4044da246540ec900f7b3"
server_key_iv: "43b3b14fa4a8808d1d4ab9dac206cb20"
server_key_version: "encryptionkey@0"
I20260619 08:56:06.943014 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:06.962680 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:06.964630 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:06.964756 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:06.968469 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37761
I20260619 08:56:06.968519 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37761 every 8 connection(s)
I20260619 08:56:06.968847 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:56:06.969763 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:06.971546 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 930
I20260619 08:56:06.971616 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 08:56:06.971751 392 external_mini_cluster.cc:1428] Setting key a56cd86a31de2e67886c7e24e32add99
I20260619 08:56:06.972208 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e04e1b71380645a8ae77963a0ebb0f5d: Bootstrap starting.
I20260619 08:56:06.972944 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e04e1b71380645a8ae77963a0ebb0f5d: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:06.973265 998 log.cc:824] T 00000000000000000000000000000000 P e04e1b71380645a8ae77963a0ebb0f5d: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:06.973906 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e04e1b71380645a8ae77963a0ebb0f5d: No bootstrap required, opened a new log
I20260619 08:56:06.976138 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P e04e1b71380645a8ae77963a0ebb0f5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04e1b71380645a8ae77963a0ebb0f5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37761 } }
I20260619 08:56:06.976310 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P e04e1b71380645a8ae77963a0ebb0f5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:06.976339 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P e04e1b71380645a8ae77963a0ebb0f5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e04e1b71380645a8ae77963a0ebb0f5d, State: Initialized, Role: FOLLOWER
I20260619 08:56:06.976470 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P e04e1b71380645a8ae77963a0ebb0f5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04e1b71380645a8ae77963a0ebb0f5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37761 } }
I20260619 08:56:06.976541 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P e04e1b71380645a8ae77963a0ebb0f5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:56:06.976576 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P e04e1b71380645a8ae77963a0ebb0f5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:56:06.976619 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e04e1b71380645a8ae77963a0ebb0f5d [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:06.977218 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P e04e1b71380645a8ae77963a0ebb0f5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04e1b71380645a8ae77963a0ebb0f5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37761 } }
I20260619 08:56:06.977317 998 leader_election.cc:302] T 00000000000000000000000000000000 P e04e1b71380645a8ae77963a0ebb0f5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e04e1b71380645a8ae77963a0ebb0f5d; no voters:
I20260619 08:56:06.977483 998 leader_election.cc:288] T 00000000000000000000000000000000 P e04e1b71380645a8ae77963a0ebb0f5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:56:06.977520 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e04e1b71380645a8ae77963a0ebb0f5d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:06.977625 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P e04e1b71380645a8ae77963a0ebb0f5d [term 1 LEADER]: Becoming Leader. State: Replica: e04e1b71380645a8ae77963a0ebb0f5d, State: Running, Role: LEADER
I20260619 08:56:06.977751 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P e04e1b71380645a8ae77963a0ebb0f5d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:56:06.977772 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P e04e1b71380645a8ae77963a0ebb0f5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04e1b71380645a8ae77963a0ebb0f5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37761 } }
I20260619 08:56:06.978117 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P e04e1b71380645a8ae77963a0ebb0f5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e04e1b71380645a8ae77963a0ebb0f5d. Latest consensus state: current_term: 1 leader_uuid: "e04e1b71380645a8ae77963a0ebb0f5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04e1b71380645a8ae77963a0ebb0f5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37761 } } }
I20260619 08:56:06.978132 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P e04e1b71380645a8ae77963a0ebb0f5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e04e1b71380645a8ae77963a0ebb0f5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e04e1b71380645a8ae77963a0ebb0f5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37761 } } }
I20260619 08:56:06.978302 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P e04e1b71380645a8ae77963a0ebb0f5d [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:06.978313 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P e04e1b71380645a8ae77963a0ebb0f5d [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:06.978570 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 19 08:56:06 dist-test-slave-h4g5 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:06.979161 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:56:06.980836 1012 catalog_manager.cc:1347] Generated new cluster ID: 1b1b11f299fa45c69f6893b2d3756281
I20260619 08:56:06.980893 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:56:06.982834 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:06.972294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53108 (local address 127.0.98.62:37761)
0619 08:56:06.972481 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:06.972490 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:06.972522 (+ 32us) server_negotiation.cc:407] Connection header received
0619 08:56:06.973091 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:06.973105 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:06.973192 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:06.973352 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:06.973950 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:06.975328 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:06.976061 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:06.976682 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:06.976787 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:06.979622 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:06.979627 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:06.979627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:06.979639 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:06.981447 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:06.981769 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:06.981772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:06.981773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:06.981854 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:06.982028 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:06.982028 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:06.982028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:06.982259 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:06.982415 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:06.982491 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 08:56:06.982634 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":90,"threads_started":1}
I20260619 08:56:06.985394 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:56:06.986063 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:56:06.994560 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e04e1b71380645a8ae77963a0ebb0f5d: Generated new TSK 0
I20260619 08:56:06.994777 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 08:56:07.020888 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37761
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:07.101938 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:07.102149 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:07.102174 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:07.103350 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:07.103396 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:07.103422 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 08:56:07.104943 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37761
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:07.105175 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:07.105400 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:07.110119 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:07.110277 1024 server_base.cc:1034] running on GCE node
W20260619 08:56:07.110122 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:07.110139 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:07.110544 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:07.110746 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:07.111912 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781859367111895 us; error 29 us; skew 500 ppm
Jun 19 08:56:07 dist-test-slave-h4g5 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:07.114239 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 08:56:07.114698 1024 webserver.cc:460] Webserver started at http://127.0.98.1:39927/ using document root <none> and password file <none>
I20260619 08:56:07.114897 1024 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:07.114955 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:07.115068 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:07.116115 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-0/data/instance:
uuid: "84e34f0c4c2544a49ed8bf73fefb005b"
format_stamp: "Formatted at 2026-06-19 08:56:07 on dist-test-slave-h4g5"
server_key: "ebf50567cad5602c05972c26b0f7bff1"
server_key_iv: "9754017ab247bc8ebbda1d3519e5c4fd"
server_key_version: "encryptionkey@0"
I20260619 08:56:07.116422 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-0/wal/instance:
uuid: "84e34f0c4c2544a49ed8bf73fefb005b"
format_stamp: "Formatted at 2026-06-19 08:56:07 on dist-test-slave-h4g5"
server_key: "ebf50567cad5602c05972c26b0f7bff1"
server_key_iv: "9754017ab247bc8ebbda1d3519e5c4fd"
server_key_version: "encryptionkey@0"
I20260619 08:56:07.117575 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:07.118188 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:07.118337 1024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:07.118417 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-0/wal
uuid: "84e34f0c4c2544a49ed8bf73fefb005b"
format_stamp: "Formatted at 2026-06-19 08:56:07 on dist-test-slave-h4g5"
server_key: "ebf50567cad5602c05972c26b0f7bff1"
server_key_iv: "9754017ab247bc8ebbda1d3519e5c4fd"
server_key_version: "encryptionkey@0"
I20260619 08:56:07.118475 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:07.140967 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:07.142731 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:07.142870 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:07.143100 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:07.143394 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:07.143427 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:07.143456 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:07.143473 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:07.149008 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45483
I20260619 08:56:07.149070 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45483 every 8 connection(s)
I20260619 08:56:07.149395 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 08:56:07 dist-test-slave-h4g5 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:07.155552 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 1024
I20260619 08:56:07.155642 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-0/wal/instance
I20260619 08:56:07.155812 392 external_mini_cluster.cc:1428] Setting key c1df2f4de0ff4a062fbd060c9add95db
I20260619 08:56:07.158269 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:07.150042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48513 (local address 127.0.98.62:37761)
0619 08:56:07.150084 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:07.150086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:07.150296 (+ 210us) server_negotiation.cc:407] Connection header received
0619 08:56:07.150786 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:07.150787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:07.150799 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:07.150856 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:07.151789 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.152363 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.153345 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.153551 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.153607 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:07.155584 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:07.155585 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:07.155586 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:07.155594 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:07.156983 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:07.157358 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:07.157359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:07.157360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:07.157412 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:07.157623 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:07.157624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:07.157625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:07.157737 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:07.157793 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:07.158081 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 08:56:07.158175 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 08:56:07.158406 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:07.150026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37761 (local address 127.0.98.1:48513)
0619 08:56:07.150176 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:07.150213 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:07.150691 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:07.150929 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:07.150933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:07.151489 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:07.151690 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:07.151694 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.152460 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.152461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:07.153263 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:07.153265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.153619 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.153620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:07.153744 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:07.154190 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:07.154198 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:07.155485 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:07.157061 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:07.157068 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:07.157070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:07.157291 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:07.157465 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:07.157466 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:07.157466 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:07.157567 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:07.157792 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:07.157798 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:07.157931 (+ 133us) client_negotiation.cc:769] Sending connection context
0619 08:56:07.158020 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 08:56:07.158186 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
I20260619 08:56:07.158854 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37761
I20260619 08:56:07.158941 1153 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:07.159111 1153 heartbeater.cc:511] Master 127.0.98.62:37761 requested a full tablet report, sending...
I20260619 08:56:07.159637 962 ts_manager.cc:194] Registered new tserver with Master: 84e34f0c4c2544a49ed8bf73fefb005b (127.0.98.1:45483)
I20260619 08:56:07.160297 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48513
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 08:56:07.179450 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37761
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:07.258689 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:07.258898 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:07.258921 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:07.260128 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:07.260172 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:07.260200 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 08:56:07.261618 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37761
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:07.261852 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:07.262055 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:07.266882 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:07.266984 1158 server_base.cc:1034] running on GCE node
W20260619 08:56:07.266992 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:07.266889 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:07.267307 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:07.267540 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:07.268676 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781859367268660 us; error 28 us; skew 500 ppm
Jun 19 08:56:07 dist-test-slave-h4g5 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:07.270970 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 08:56:07.271405 1158 webserver.cc:460] Webserver started at http://127.0.98.2:46217/ using document root <none> and password file <none>
I20260619 08:56:07.271610 1158 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:07.271656 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:07.271776 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:07.272923 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-1/data/instance:
uuid: "4599221b9daa490d81c6d4639f48d04e"
format_stamp: "Formatted at 2026-06-19 08:56:07 on dist-test-slave-h4g5"
server_key: "26efb31874fda5b9da6fcb6b718ead8a"
server_key_iv: "e478962828ac1143ca05049fc4011945"
server_key_version: "encryptionkey@0"
I20260619 08:56:07.273257 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4599221b9daa490d81c6d4639f48d04e"
format_stamp: "Formatted at 2026-06-19 08:56:07 on dist-test-slave-h4g5"
server_key: "26efb31874fda5b9da6fcb6b718ead8a"
server_key_iv: "e478962828ac1143ca05049fc4011945"
server_key_version: "encryptionkey@0"
I20260619 08:56:07.274469 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 08:56:07.275130 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:07.275281 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 08:56:07.275360 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-1/wal
uuid: "4599221b9daa490d81c6d4639f48d04e"
format_stamp: "Formatted at 2026-06-19 08:56:07 on dist-test-slave-h4g5"
server_key: "26efb31874fda5b9da6fcb6b718ead8a"
server_key_iv: "e478962828ac1143ca05049fc4011945"
server_key_version: "encryptionkey@0"
I20260619 08:56:07.275417 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:07.286024 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:07.287807 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:07.287946 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:07.288323 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:07.288667 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:07.288703 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:07.288733 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:07.288749 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:07.294154 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35873
I20260619 08:56:07.294209 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35873 every 8 connection(s)
I20260619 08:56:07.294504 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 08:56:07 dist-test-slave-h4g5 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:07.303270 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:07.295180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43999 (local address 127.0.98.62:37761)
0619 08:56:07.295232 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:07.295234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:07.295430 (+ 196us) server_negotiation.cc:407] Connection header received
0619 08:56:07.295932 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:07.295933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:07.295942 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:07.295981 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:07.296906 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.297413 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.298410 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.298619 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.298683 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:07.300794 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:07.300796 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:07.300796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:07.300804 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:07.302025 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:07.302387 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:07.302388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:07.302388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:07.302437 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:07.302669 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:07.302670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:07.302670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:07.302774 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:07.302836 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:07.303092 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 08:56:07.303177 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 08:56:07.303439 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:07.295153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37761 (local address 127.0.98.2:43999)
0619 08:56:07.295309 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:07.295352 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:07.295828 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:07.296038 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:07.296042 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:07.296639 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:07.296839 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:07.296844 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.297480 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.297481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:07.298316 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:07.298318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.298692 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.298693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:07.298796 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:07.299251 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:07.299259 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:07.300700 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:07.302100 (+ 1400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:07.302108 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:07.302110 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:07.302321 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:07.302488 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:07.302489 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:07.302489 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:07.302614 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:07.302833 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:07.302840 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:07.302977 (+ 137us) client_negotiation.cc:769] Sending connection context
0619 08:56:07.303065 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 08:56:07.303222 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260619 08:56:07.303927 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37761
I20260619 08:56:07.304023 1287 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:07.304229 1287 heartbeater.cc:511] Master 127.0.98.62:37761 requested a full tablet report, sending...
I20260619 08:56:07.304244 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 1158
I20260619 08:56:07.304318 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-1/wal/instance
I20260619 08:56:07.304464 392 external_mini_cluster.cc:1428] Setting key 0cc599325ed78f93f045e1415ba487a0
I20260619 08:56:07.304625 962 ts_manager.cc:194] Registered new tserver with Master: 4599221b9daa490d81c6d4639f48d04e (127.0.98.2:35873)
I20260619 08:56:07.305092 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43999
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 08:56:07.330523 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37761
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:07.410140 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:07.410360 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:07.410384 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:07.411607 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:07.411652 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:07.411679 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 08:56:07.413198 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37761
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:07.413442 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:07.413673 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:07.418695 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:07.418705 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:07.418900 1292 server_base.cc:1034] running on GCE node
W20260619 08:56:07.418705 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:07.419117 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:07.419353 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:07.420480 1292 hybrid_clock.cc:648] HybridClock initialized: now 1781859367420471 us; error 27 us; skew 500 ppm
Jun 19 08:56:07 dist-test-slave-h4g5 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:07.422873 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 08:56:07.423303 1292 webserver.cc:460] Webserver started at http://127.0.98.3:34461/ using document root <none> and password file <none>
I20260619 08:56:07.423501 1292 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:07.423540 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:07.423629 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:07.424688 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-2/data/instance:
uuid: "436b0f0f594145d8a9ee49b92f864b70"
format_stamp: "Formatted at 2026-06-19 08:56:07 on dist-test-slave-h4g5"
server_key: "6543ece14560b4b5c6feaeb46c3d411d"
server_key_iv: "00d617c738711e0c3e24bd66c6d2e9ec"
server_key_version: "encryptionkey@0"
I20260619 08:56:07.424970 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-2/wal/instance:
uuid: "436b0f0f594145d8a9ee49b92f864b70"
format_stamp: "Formatted at 2026-06-19 08:56:07 on dist-test-slave-h4g5"
server_key: "6543ece14560b4b5c6feaeb46c3d411d"
server_key_iv: "00d617c738711e0c3e24bd66c6d2e9ec"
server_key_version: "encryptionkey@0"
I20260619 08:56:07.426123 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:56:07.426733 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:07.426903 1292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:07.426985 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-2/wal
uuid: "436b0f0f594145d8a9ee49b92f864b70"
format_stamp: "Formatted at 2026-06-19 08:56:07 on dist-test-slave-h4g5"
server_key: "6543ece14560b4b5c6feaeb46c3d411d"
server_key_iv: "00d617c738711e0c3e24bd66c6d2e9ec"
server_key_version: "encryptionkey@0"
I20260619 08:56:07.427039 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:07.449469 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:07.451552 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:07.451664 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:07.451862 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:07.452189 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:07.452227 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:07.452257 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:07.452281 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:07.457573 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38069
I20260619 08:56:07.457613 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38069 every 8 connection(s)
I20260619 08:56:07.457924 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 08:56:07 dist-test-slave-h4g5 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:07.465294 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 1292
I20260619 08:56:07.465378 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859361540516-392-0/minicluster-data/ts-2/wal/instance
I20260619 08:56:07.465526 392 external_mini_cluster.cc:1428] Setting key 4f69c6cb6f4a9e9fecd4849e46176b37
I20260619 08:56:07.466970 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:07.458629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56929 (local address 127.0.98.62:37761)
0619 08:56:07.458710 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:07.458713 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:07.458886 (+ 173us) server_negotiation.cc:407] Connection header received
0619 08:56:07.459409 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:07.459411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:07.459424 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:07.459475 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:07.460472 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.460980 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.461966 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.462167 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.462230 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:07.464269 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:07.464271 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:07.464271 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:07.464280 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:07.465531 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:07.466063 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:07.466065 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:07.466065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:07.466119 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:07.466365 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:07.466366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:07.466366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:07.466478 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:07.466534 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:07.466801 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 08:56:07.466881 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 08:56:07.467150 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:07.458590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37761 (local address 127.0.98.3:56929)
0619 08:56:07.458764 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:07.458807 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:07.459309 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:07.459548 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:07.459552 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:07.460170 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:07.460380 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:07.460385 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.461069 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.461070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:07.461884 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:07.461886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.462245 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.462246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:07.462369 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:07.462844 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:07.462853 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:07.464175 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:07.465715 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:07.465723 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:07.465725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:07.466002 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:07.466172 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:07.466173 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:07.466173 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:07.466304 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:07.466532 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:07.466538 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:07.466675 (+ 137us) client_negotiation.cc:769] Sending connection context
0619 08:56:07.466780 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 08:56:07.466935 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260619 08:56:07.467541 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37761
I20260619 08:56:07.467628 1421 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:07.467797 1421 heartbeater.cc:511] Master 127.0.98.62:37761 requested a full tablet report, sending...
I20260619 08:56:07.468142 962 ts_manager.cc:194] Registered new tserver with Master: 436b0f0f594145d8a9ee49b92f864b70 (127.0.98.3:38069)
I20260619 08:56:07.468509 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56929
I20260619 08:56:07.477329 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:56:07.485399 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:07.479024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53112 (local address 127.0.98.62:37761)
0619 08:56:07.479101 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:07.479103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:07.479163 (+ 60us) server_negotiation.cc:407] Connection header received
0619 08:56:07.479218 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:07.479219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:07.479229 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:07.479295 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:07.479885 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.480491 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.481246 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.481431 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.481486 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:07.482830 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:07.482832 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:07.482833 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:07.482841 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:07.484238 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:07.484583 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:07.484584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:07.484584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:07.484644 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:07.484820 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:07.484821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:07.484822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:07.484979 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:07.485084 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:07.485208 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 08:56:07.485297 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 08:56:07.487756 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:56:07.489445 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:56:07.495823 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:07.491987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58656 (local address 127.0.98.1:45483)
0619 08:56:07.492212 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:07.492216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:07.492288 (+ 72us) server_negotiation.cc:407] Connection header received
0619 08:56:07.492378 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:07.492380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:07.492393 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:07.492466 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:07.492669 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.493490 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.494889 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.495444 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.495512 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:07.495586 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:07.495705 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 08:56:07.495746 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":153,"threads_started":1}
I20260619 08:56:07.495828 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:07.491747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45483 (local address 127.0.0.1:58656)
0619 08:56:07.492198 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:07.492239 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:07.492306 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:07.492491 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:07.492494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:07.492497 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:07.492572 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:07.492573 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.493585 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.493586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:07.494807 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:07.494809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.495521 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.495522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:07.495605 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:07.495619 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 08:56:07.495683 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 08:56:07.495737 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":65,"threads_started":1}
I20260619 08:56:07.496146 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:07.491751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38069 (local address 127.0.0.1:49332)
0619 08:56:07.492374 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:07.492380 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:07.492412 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:07.492629 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:07.492630 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:07.492632 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:07.492681 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:07.492684 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.493926 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.493927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:07.494976 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:07.494978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.495903 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.495903 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:07.495954 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:07.495957 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:56:07.495996 (+ 39us) client_negotiation.cc:240] Negotiation successful
0619 08:56:07.496046 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":70,"threads_started":1}
I20260619 08:56:07.496344 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:07.491916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56370 (local address 127.0.98.2:35873)
0619 08:56:07.492071 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:07.492075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:07.492521 (+ 446us) server_negotiation.cc:407] Connection header received
0619 08:56:07.492625 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:07.492628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:07.492644 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:07.492712 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:07.492906 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.493971 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.494919 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.495646 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.495716 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:07.495778 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:07.496226 (+ 448us) server_negotiation.cc:299] Negotiation successful
0619 08:56:07.496271 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
I20260619 08:56:07.496435 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:07.492000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49332 (local address 127.0.98.3:38069)
0619 08:56:07.492176 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:07.492179 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:07.492428 (+ 249us) server_negotiation.cc:407] Connection header received
0619 08:56:07.492466 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:07.492469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:07.492483 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:07.492548 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:07.492749 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.493825 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.495069 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.495767 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.495823 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:07.495871 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:07.496323 (+ 452us) server_negotiation.cc:299] Negotiation successful
0619 08:56:07.496363 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":96,"threads_started":1}
I20260619 08:56:07.496572 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:07.491760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35873 (local address 127.0.0.1:56370)
0619 08:56:07.492455 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:07.492462 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:07.492550 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:07.492791 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:07.492792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:07.492794 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:07.492832 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:07.492833 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.494054 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.494055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:07.494854 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:07.494857 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.495814 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.495815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:07.495892 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:07.495896 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:56:07.496453 (+ 557us) client_negotiation.cc:240] Negotiation successful
0619 08:56:07.496496 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":53,"threads_started":1}
I20260619 08:56:07.496687 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 4423ea5753544441b853ab67d078a51b (DEFAULT_TABLE table=test-table [id=8d592dc0cb834abb843250bd411f0285]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:07.497094 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4423ea5753544441b853ab67d078a51b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:07.497149 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 4423ea5753544441b853ab67d078a51b (DEFAULT_TABLE table=test-table [id=8d592dc0cb834abb843250bd411f0285]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:07.497253 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 4423ea5753544441b853ab67d078a51b (DEFAULT_TABLE table=test-table [id=8d592dc0cb834abb843250bd411f0285]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:07.497464 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4423ea5753544441b853ab67d078a51b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:07.497514 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4423ea5753544441b853ab67d078a51b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:07.499514 1435 tablet_bootstrap.cc:492] T 4423ea5753544441b853ab67d078a51b P 84e34f0c4c2544a49ed8bf73fefb005b: Bootstrap starting.
I20260619 08:56:07.500344 1435 tablet_bootstrap.cc:654] T 4423ea5753544441b853ab67d078a51b P 84e34f0c4c2544a49ed8bf73fefb005b: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:07.500648 1436 tablet_bootstrap.cc:492] T 4423ea5753544441b853ab67d078a51b P 4599221b9daa490d81c6d4639f48d04e: Bootstrap starting.
I20260619 08:56:07.500919 1435 log.cc:824] T 4423ea5753544441b853ab67d078a51b P 84e34f0c4c2544a49ed8bf73fefb005b: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:07.501284 1436 tablet_bootstrap.cc:654] T 4423ea5753544441b853ab67d078a51b P 4599221b9daa490d81c6d4639f48d04e: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:07.501655 1435 tablet_bootstrap.cc:492] T 4423ea5753544441b853ab67d078a51b P 84e34f0c4c2544a49ed8bf73fefb005b: No bootstrap required, opened a new log
I20260619 08:56:07.501720 1435 ts_tablet_manager.cc:1397] T 4423ea5753544441b853ab67d078a51b P 84e34f0c4c2544a49ed8bf73fefb005b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:56:07.502254 1436 log.cc:824] T 4423ea5753544441b853ab67d078a51b P 4599221b9daa490d81c6d4639f48d04e: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:07.502604 1437 tablet_bootstrap.cc:492] T 4423ea5753544441b853ab67d078a51b P 436b0f0f594145d8a9ee49b92f864b70: Bootstrap starting.
I20260619 08:56:07.503275 1437 tablet_bootstrap.cc:654] T 4423ea5753544441b853ab67d078a51b P 436b0f0f594145d8a9ee49b92f864b70: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:07.503741 1437 log.cc:824] T 4423ea5753544441b853ab67d078a51b P 436b0f0f594145d8a9ee49b92f864b70: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:07.503816 1435 raft_consensus.cc:348] T 4423ea5753544441b853ab67d078a51b P 84e34f0c4c2544a49ed8bf73fefb005b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e34f0c4c2544a49ed8bf73fefb005b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45483 } } peers { permanent_uuid: "4599221b9daa490d81c6d4639f48d04e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35873 } } peers { permanent_uuid: "436b0f0f594145d8a9ee49b92f864b70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38069 } }
I20260619 08:56:07.504009 1435 raft_consensus.cc:374] T 4423ea5753544441b853ab67d078a51b P 84e34f0c4c2544a49ed8bf73fefb005b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:07.504076 1435 raft_consensus.cc:729] T 4423ea5753544441b853ab67d078a51b P 84e34f0c4c2544a49ed8bf73fefb005b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84e34f0c4c2544a49ed8bf73fefb005b, State: Initialized, Role: FOLLOWER
I20260619 08:56:07.504295 1436 tablet_bootstrap.cc:492] T 4423ea5753544441b853ab67d078a51b P 4599221b9daa490d81c6d4639f48d04e: No bootstrap required, opened a new log
I20260619 08:56:07.504261 1435 consensus_queue.cc:260] T 4423ea5753544441b853ab67d078a51b P 84e34f0c4c2544a49ed8bf73fefb005b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e34f0c4c2544a49ed8bf73fefb005b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45483 } } peers { permanent_uuid: "4599221b9daa490d81c6d4639f48d04e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35873 } } peers { permanent_uuid: "436b0f0f594145d8a9ee49b92f864b70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38069 } }
I20260619 08:56:07.504360 1436 ts_tablet_manager.cc:1397] T 4423ea5753544441b853ab67d078a51b P 4599221b9daa490d81c6d4639f48d04e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 08:56:07.504528 1437 tablet_bootstrap.cc:492] T 4423ea5753544441b853ab67d078a51b P 436b0f0f594145d8a9ee49b92f864b70: No bootstrap required, opened a new log
I20260619 08:56:07.504549 1435 ts_tablet_manager.cc:1428] T 4423ea5753544441b853ab67d078a51b P 84e34f0c4c2544a49ed8bf73fefb005b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:56:07.504606 1437 ts_tablet_manager.cc:1397] T 4423ea5753544441b853ab67d078a51b P 436b0f0f594145d8a9ee49b92f864b70: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:56:07.504673 1153 heartbeater.cc:503] Master 127.0.98.62:37761 was elected leader, sending a full tablet report...
I20260619 08:56:07.505993 1436 raft_consensus.cc:348] T 4423ea5753544441b853ab67d078a51b P 4599221b9daa490d81c6d4639f48d04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e34f0c4c2544a49ed8bf73fefb005b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45483 } } peers { permanent_uuid: "4599221b9daa490d81c6d4639f48d04e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35873 } } peers { permanent_uuid: "436b0f0f594145d8a9ee49b92f864b70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38069 } }
I20260619 08:56:07.506141 1436 raft_consensus.cc:374] T 4423ea5753544441b853ab67d078a51b P 4599221b9daa490d81c6d4639f48d04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:07.506173 1436 raft_consensus.cc:729] T 4423ea5753544441b853ab67d078a51b P 4599221b9daa490d81c6d4639f48d04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4599221b9daa490d81c6d4639f48d04e, State: Initialized, Role: FOLLOWER
I20260619 08:56:07.506287 1436 consensus_queue.cc:260] T 4423ea5753544441b853ab67d078a51b P 4599221b9daa490d81c6d4639f48d04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e34f0c4c2544a49ed8bf73fefb005b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45483 } } peers { permanent_uuid: "4599221b9daa490d81c6d4639f48d04e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35873 } } peers { permanent_uuid: "436b0f0f594145d8a9ee49b92f864b70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38069 } }
I20260619 08:56:07.506374 1437 raft_consensus.cc:348] T 4423ea5753544441b853ab67d078a51b P 436b0f0f594145d8a9ee49b92f864b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e34f0c4c2544a49ed8bf73fefb005b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45483 } } peers { permanent_uuid: "4599221b9daa490d81c6d4639f48d04e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35873 } } peers { permanent_uuid: "436b0f0f594145d8a9ee49b92f864b70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38069 } }
I20260619 08:56:07.506563 1287 heartbeater.cc:503] Master 127.0.98.62:37761 was elected leader, sending a full tablet report...
I20260619 08:56:07.506558 1437 raft_consensus.cc:374] T 4423ea5753544441b853ab67d078a51b P 436b0f0f594145d8a9ee49b92f864b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:07.506605 1437 raft_consensus.cc:729] T 4423ea5753544441b853ab67d078a51b P 436b0f0f594145d8a9ee49b92f864b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 436b0f0f594145d8a9ee49b92f864b70, State: Initialized, Role: FOLLOWER
I20260619 08:56:07.506727 1437 consensus_queue.cc:260] T 4423ea5753544441b853ab67d078a51b P 436b0f0f594145d8a9ee49b92f864b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e34f0c4c2544a49ed8bf73fefb005b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45483 } } peers { permanent_uuid: "4599221b9daa490d81c6d4639f48d04e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35873 } } peers { permanent_uuid: "436b0f0f594145d8a9ee49b92f864b70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38069 } }
I20260619 08:56:07.506801 1436 ts_tablet_manager.cc:1428] T 4423ea5753544441b853ab67d078a51b P 4599221b9daa490d81c6d4639f48d04e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 08:56:07.506999 1437 ts_tablet_manager.cc:1428] T 4423ea5753544441b853ab67d078a51b P 436b0f0f594145d8a9ee49b92f864b70: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:56:07.507166 1421 heartbeater.cc:503] Master 127.0.98.62:37761 was elected leader, sending a full tablet report...
W20260619 08:56:07.545071 1288 tablet.cc:2369] T 4423ea5753544441b853ab67d078a51b P 4599221b9daa490d81c6d4639f48d04e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 08:56:07.650197 1154 tablet.cc:2369] T 4423ea5753544441b853ab67d078a51b P 84e34f0c4c2544a49ed8bf73fefb005b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:56:07.695250 1441 raft_consensus.cc:482] T 4423ea5753544441b853ab67d078a51b P 84e34f0c4c2544a49ed8bf73fefb005b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:56:07.695405 1441 raft_consensus.cc:504] T 4423ea5753544441b853ab67d078a51b P 84e34f0c4c2544a49ed8bf73fefb005b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e34f0c4c2544a49ed8bf73fefb005b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45483 } } peers { permanent_uuid: "4599221b9daa490d81c6d4639f48d04e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35873 } } peers { permanent_uuid: "436b0f0f594145d8a9ee49b92f864b70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38069 } }
I20260619 08:56:07.695811 1441 leader_election.cc:288] T 4423ea5753544441b853ab67d078a51b P 84e34f0c4c2544a49ed8bf73fefb005b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4599221b9daa490d81c6d4639f48d04e (127.0.98.2:35873), 436b0f0f594145d8a9ee49b92f864b70 (127.0.98.3:38069)
I20260619 08:56:07.699716 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:07.696009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35873 (local address 127.0.98.1:58817)
0619 08:56:07.696228 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:07.696240 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:07.696316 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:07.696506 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:07.696506 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:07.696508 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:07.696619 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:07.696620 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.697538 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.697538 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:07.698796 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:07.698800 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.699467 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.699468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:07.699541 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:07.699546 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 08:56:07.699593 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 08:56:07.699642 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":64,"thread_start_us":149,"threads_started":1}
I20260619 08:56:07.699728 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:07.696113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58817 (local address 127.0.98.2:35873)
0619 08:56:07.696162 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:07.696164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:07.696352 (+ 188us) server_negotiation.cc:407] Connection header received
0619 08:56:07.696397 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:07.696398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:07.696413 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:07.696489 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:07.696689 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.697431 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.698904 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.699384 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.699457 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:07.699468 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:07.699613 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 08:56:07.699656 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 08:56:07.700165 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:07.696015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38069 (local address 127.0.98.1:38567)
0619 08:56:07.696331 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:07.696340 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:07.696394 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:07.696527 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:07.696528 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:07.696529 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:07.696612 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:07.696614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.697521 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.697522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:07.699151 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:07.699153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.699883 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.699884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:07.699961 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:07.699965 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:56:07.700015 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 08:56:07.700057 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":64,"thread_start_us":160,"threads_started":1}
I20260619 08:56:07.700207 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:07.696159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38567 (local address 127.0.98.3:38069)
0619 08:56:07.696205 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:07.696207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:07.696424 (+ 217us) server_negotiation.cc:407] Connection header received
0619 08:56:07.696451 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:07.696452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:07.696464 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:07.696514 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:07.696686 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.697431 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.699257 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.699774 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.699837 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:07.699846 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:07.700070 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 08:56:07.700142 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 08:56:07.700163 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4423ea5753544441b853ab67d078a51b" candidate_uuid: "84e34f0c4c2544a49ed8bf73fefb005b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4599221b9daa490d81c6d4639f48d04e" is_pre_election: true
I20260619 08:56:07.700348 1241 raft_consensus.cc:2454] T 4423ea5753544441b853ab67d078a51b P 4599221b9daa490d81c6d4639f48d04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84e34f0c4c2544a49ed8bf73fefb005b in term 0.
I20260619 08:56:07.700459 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4423ea5753544441b853ab67d078a51b" candidate_uuid: "84e34f0c4c2544a49ed8bf73fefb005b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "436b0f0f594145d8a9ee49b92f864b70" is_pre_election: true
I20260619 08:56:07.700610 1043 leader_election.cc:302] T 4423ea5753544441b853ab67d078a51b P 84e34f0c4c2544a49ed8bf73fefb005b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4599221b9daa490d81c6d4639f48d04e, 84e34f0c4c2544a49ed8bf73fefb005b; no voters:
I20260619 08:56:07.700615 1375 raft_consensus.cc:2454] T 4423ea5753544441b853ab67d078a51b P 436b0f0f594145d8a9ee49b92f864b70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84e34f0c4c2544a49ed8bf73fefb005b in term 0.
I20260619 08:56:07.700769 1441 raft_consensus.cc:2781] T 4423ea5753544441b853ab67d078a51b P 84e34f0c4c2544a49ed8bf73fefb005b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:56:07.700839 1441 raft_consensus.cc:482] T 4423ea5753544441b853ab67d078a51b P 84e34f0c4c2544a49ed8bf73fefb005b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:56:07.700861 1441 raft_consensus.cc:3037] T 4423ea5753544441b853ab67d078a51b P 84e34f0c4c2544a49ed8bf73fefb005b [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:07.701534 1441 raft_consensus.cc:504] T 4423ea5753544441b853ab67d078a51b P 84e34f0c4c2544a49ed8bf73fefb005b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e34f0c4c2544a49ed8bf73fefb005b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45483 } } peers { permanent_uuid: "4599221b9daa490d81c6d4639f48d04e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35873 } } peers { permanent_uuid: "436b0f0f594145d8a9ee49b92f864b70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38069 } }
I20260619 08:56:07.701663 1441 leader_election.cc:288] T 4423ea5753544441b853ab67d078a51b P 84e34f0c4c2544a49ed8bf73fefb005b [CANDIDATE]: Term 1 election: Requested vote from peers 4599221b9daa490d81c6d4639f48d04e (127.0.98.2:35873), 436b0f0f594145d8a9ee49b92f864b70 (127.0.98.3:38069)
I20260619 08:56:07.701830 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4423ea5753544441b853ab67d078a51b" candidate_uuid: "84e34f0c4c2544a49ed8bf73fefb005b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "436b0f0f594145d8a9ee49b92f864b70"
I20260619 08:56:07.701869 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4423ea5753544441b853ab67d078a51b" candidate_uuid: "84e34f0c4c2544a49ed8bf73fefb005b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4599221b9daa490d81c6d4639f48d04e"
I20260619 08:56:07.701890 1375 raft_consensus.cc:3037] T 4423ea5753544441b853ab67d078a51b P 436b0f0f594145d8a9ee49b92f864b70 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:07.701954 1241 raft_consensus.cc:3037] T 4423ea5753544441b853ab67d078a51b P 4599221b9daa490d81c6d4639f48d04e [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:07.702574 1375 raft_consensus.cc:2454] T 4423ea5753544441b853ab67d078a51b P 436b0f0f594145d8a9ee49b92f864b70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84e34f0c4c2544a49ed8bf73fefb005b in term 1.
I20260619 08:56:07.702737 1041 leader_election.cc:302] T 4423ea5753544441b853ab67d078a51b P 84e34f0c4c2544a49ed8bf73fefb005b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 436b0f0f594145d8a9ee49b92f864b70, 84e34f0c4c2544a49ed8bf73fefb005b; no voters:
I20260619 08:56:07.702790 1241 raft_consensus.cc:2454] T 4423ea5753544441b853ab67d078a51b P 4599221b9daa490d81c6d4639f48d04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84e34f0c4c2544a49ed8bf73fefb005b in term 1.
I20260619 08:56:07.702840 1441 raft_consensus.cc:2781] T 4423ea5753544441b853ab67d078a51b P 84e34f0c4c2544a49ed8bf73fefb005b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:07.703004 1441 raft_consensus.cc:686] T 4423ea5753544441b853ab67d078a51b P 84e34f0c4c2544a49ed8bf73fefb005b [term 1 LEADER]: Becoming Leader. State: Replica: 84e34f0c4c2544a49ed8bf73fefb005b, State: Running, Role: LEADER
I20260619 08:56:07.703094 1441 consensus_queue.cc:237] T 4423ea5753544441b853ab67d078a51b P 84e34f0c4c2544a49ed8bf73fefb005b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e34f0c4c2544a49ed8bf73fefb005b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45483 } } peers { permanent_uuid: "4599221b9daa490d81c6d4639f48d04e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35873 } } peers { permanent_uuid: "436b0f0f594145d8a9ee49b92f864b70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38069 } }
I20260619 08:56:07.703907 960 catalog_manager.cc:5495] T 4423ea5753544441b853ab67d078a51b P 84e34f0c4c2544a49ed8bf73fefb005b reported cstate change: term changed from 0 to 1, leader changed from <none> to 84e34f0c4c2544a49ed8bf73fefb005b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "84e34f0c4c2544a49ed8bf73fefb005b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e34f0c4c2544a49ed8bf73fefb005b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4599221b9daa490d81c6d4639f48d04e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "436b0f0f594145d8a9ee49b92f864b70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38069 } health_report { overall_health: UNKNOWN } } }
W20260619 08:56:07.708437 1422 tablet.cc:2369] T 4423ea5753544441b853ab67d078a51b P 436b0f0f594145d8a9ee49b92f864b70: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 08:56:07 dist-test-slave-h4g5 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 08:56:07.726051 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:07.718383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58672 (local address 127.0.98.1:45483)
0619 08:56:07.718459 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:07.718461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:07.718558 (+ 97us) server_negotiation.cc:407] Connection header received
0619 08:56:07.718630 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:07.718631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:07.718643 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:07.718689 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:07.719246 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.719715 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.720479 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.720674 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.720728 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:07.723151 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:07.723163 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:07.723163 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:07.723172 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:07.724860 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:07.725235 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:07.725238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:07.725238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:07.725299 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:07.725480 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:07.725480 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:07.725481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:07.725643 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:07.725796 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:07.725861 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 08:56:07.725948 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 08:56:07 dist-test-slave-h4g5 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859367, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 08:56:07 dist-test-slave-h4g5 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859367, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 08:56:07.747931 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:07.741293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58674 (local address 127.0.98.1:45483)
0619 08:56:07.741372 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:07.741374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:07.741441 (+ 67us) server_negotiation.cc:407] Connection header received
0619 08:56:07.741516 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:07.741517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:07.741527 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:07.741575 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:07.742104 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.742589 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.743311 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.743497 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.743551 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:07.745696 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:07.745697 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:07.745698 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:07.745706 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:07.746854 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:07.747204 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:07.747205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:07.747206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:07.747255 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:07.747414 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:07.747415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:07.747416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:07.747508 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:07.747591 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:07.747759 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 08:56:07.747836 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 08:56:07 dist-test-slave-h4g5 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 08:56:07 dist-test-slave-h4g5 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 08:56:07.769474 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:07.762917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58684 (local address 127.0.98.1:45483)
0619 08:56:07.762968 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:07.762970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:07.762981 (+ 11us) server_negotiation.cc:407] Connection header received
0619 08:56:07.763048 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:07.763049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:07.763057 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:07.763104 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:07.763602 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.764105 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.764871 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:07.765057 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:07.765111 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:07.767342 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:07.767344 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:07.767344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:07.767351 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:07.768474 (+ 1123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:07.768780 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:07.768781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:07.768781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:07.768862 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:07.769033 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:07.769034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:07.769034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:07.769127 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:07.769211 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:07.769305 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 08:56:07.769378 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260619 08:56:07.769919 1067 tablet_service.cc:2908] Rejecting scan request for tablet 4423ea5753544441b853ab67d078a51b: Uninitialized: clean time has not yet been initialized
I20260619 08:56:07.770835 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 1024
I20260619 08:56:07.775592 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 1158
I20260619 08:56:07.779718 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 1292
I20260619 08:56:07.784214 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 930
2026-06-19T08:56:07Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1997 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:56:07 dist-test-slave-h4g5 krb5kdc[1460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:56:07 dist-test-slave-h4g5 krb5kdc[1460](info): set up 2 sockets
Jun 19 08:56:07 dist-test-slave-h4g5 krb5kdc[1460](info): commencing operation
krb5kdc: starting...
W20260619 08:56:08.743585 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 08:56:08 dist-test-slave-h4g5 krb5kdc[1460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:56:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:56:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 08:56:08.855597 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38807
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:08.934338 1473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:08.934582 1473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:08.934616 1473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:08.935864 1473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:56:08.935912 1473 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 08:56:08.935930 1473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:08.935945 1473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:08.935961 1473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:56:08.935977 1473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:56:08.937647 1473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38807
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:08.937928 1473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:08.938145 1473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:08.943095 1478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:08.943166 1479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:08.943095 1481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:08.943321 1473 server_base.cc:1034] running on GCE node
I20260619 08:56:08.943648 1473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:08.943965 1473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:08.945142 1473 hybrid_clock.cc:648] HybridClock initialized: now 1781859368945115 us; error 37 us; skew 500 ppm
Jun 19 08:56:08 dist-test-slave-h4g5 krb5kdc[1460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:08.947768 1473 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 08:56:08.948275 1473 webserver.cc:460] Webserver started at http://127.0.98.62:36645/ using document root <none> and password file <none>
I20260619 08:56:08.948498 1473 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:08.948549 1473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:08.948676 1473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:08.949728 1473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "4aa76475773049539f1aaa539293fd94"
format_stamp: "Formatted at 2026-06-19 08:56:08 on dist-test-slave-h4g5"
server_key: "4068cb050874cce5604f2c2b8d7ce4f0"
server_key_iv: "f9803a5118209c5ed57d1f3fb1daf0b6"
server_key_version: "encryptionkey@0"
I20260619 08:56:08.950080 1473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "4aa76475773049539f1aaa539293fd94"
format_stamp: "Formatted at 2026-06-19 08:56:08 on dist-test-slave-h4g5"
server_key: "4068cb050874cce5604f2c2b8d7ce4f0"
server_key_iv: "f9803a5118209c5ed57d1f3fb1daf0b6"
server_key_version: "encryptionkey@0"
I20260619 08:56:08.951364 1473 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:56:08.952051 1488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:08.952240 1473 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:08.952324 1473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "4aa76475773049539f1aaa539293fd94"
format_stamp: "Formatted at 2026-06-19 08:56:08 on dist-test-slave-h4g5"
server_key: "4068cb050874cce5604f2c2b8d7ce4f0"
server_key_iv: "f9803a5118209c5ed57d1f3fb1daf0b6"
server_key_version: "encryptionkey@0"
I20260619 08:56:08.952389 1473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:08.973518 1473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:08.975498 1473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:08.975637 1473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:08.979984 1473 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38807
I20260619 08:56:08.980033 1550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38807 every 8 connection(s)
I20260619 08:56:08.980376 1473 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:56:08.980614 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 1473
I20260619 08:56:08.980695 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 08:56:08.980855 392 external_mini_cluster.cc:1428] Setting key 6a42e12f225ee6cf4a650601a756ceda
I20260619 08:56:08.981083 1551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:08.983835 1551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aa76475773049539f1aaa539293fd94: Bootstrap starting.
I20260619 08:56:08.984464 1551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4aa76475773049539f1aaa539293fd94: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:08.984731 1551 log.cc:824] T 00000000000000000000000000000000 P 4aa76475773049539f1aaa539293fd94: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:08.985462 1551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aa76475773049539f1aaa539293fd94: No bootstrap required, opened a new log
Jun 19 08:56:08 dist-test-slave-h4g5 krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:08.987862 1551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4aa76475773049539f1aaa539293fd94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa76475773049539f1aaa539293fd94" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38807 } }
I20260619 08:56:08.988008 1551 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4aa76475773049539f1aaa539293fd94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:08.988039 1551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4aa76475773049539f1aaa539293fd94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aa76475773049539f1aaa539293fd94, State: Initialized, Role: FOLLOWER
I20260619 08:56:08.988202 1551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4aa76475773049539f1aaa539293fd94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa76475773049539f1aaa539293fd94" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38807 } }
I20260619 08:56:08.988292 1551 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4aa76475773049539f1aaa539293fd94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:56:08.988323 1551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4aa76475773049539f1aaa539293fd94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:56:08.988369 1551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4aa76475773049539f1aaa539293fd94 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:08.989060 1551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4aa76475773049539f1aaa539293fd94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa76475773049539f1aaa539293fd94" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38807 } }
I20260619 08:56:08.989177 1551 leader_election.cc:302] T 00000000000000000000000000000000 P 4aa76475773049539f1aaa539293fd94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4aa76475773049539f1aaa539293fd94; no voters:
I20260619 08:56:08.989347 1551 leader_election.cc:288] T 00000000000000000000000000000000 P 4aa76475773049539f1aaa539293fd94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:56:08.989398 1556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4aa76475773049539f1aaa539293fd94 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:08.989548 1556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4aa76475773049539f1aaa539293fd94 [term 1 LEADER]: Becoming Leader. State: Replica: 4aa76475773049539f1aaa539293fd94, State: Running, Role: LEADER
I20260619 08:56:08.989603 1551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4aa76475773049539f1aaa539293fd94 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:56:08.989715 1556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4aa76475773049539f1aaa539293fd94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa76475773049539f1aaa539293fd94" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38807 } }
I20260619 08:56:08.990119 1557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aa76475773049539f1aaa539293fd94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4aa76475773049539f1aaa539293fd94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa76475773049539f1aaa539293fd94" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38807 } } }
I20260619 08:56:08.990206 1557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aa76475773049539f1aaa539293fd94 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:08.990427 1558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aa76475773049539f1aaa539293fd94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4aa76475773049539f1aaa539293fd94. Latest consensus state: current_term: 1 leader_uuid: "4aa76475773049539f1aaa539293fd94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa76475773049539f1aaa539293fd94" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38807 } } }
I20260619 08:56:08.990556 1558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aa76475773049539f1aaa539293fd94 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:08.990731 1565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:56:08.991251 1565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:56:08.991688 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:08.981344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36962 (local address 127.0.98.62:38807)
0619 08:56:08.981517 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:08.981525 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:08.981549 (+ 24us) server_negotiation.cc:407] Connection header received
0619 08:56:08.982074 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:08.982081 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:08.982131 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:08.982254 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:08.982851 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:08.983728 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:08.984549 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:08.985024 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:08.985136 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:08.987825 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:08.987829 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:08.987830 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:08.987842 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:08.989842 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:08.990327 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:08.990330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:08.990331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:08.990453 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:08.990664 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:08.990665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:08.990665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:08.990977 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:08.991113 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:08.991366 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 08:56:08.991493 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":84,"threads_started":1}
I20260619 08:56:08.993083 1565 catalog_manager.cc:1347] Generated new cluster ID: f5c000702def4561adb9c8eecc9b6542
I20260619 08:56:08.993125 1565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:56:09.013223 1565 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:56:09.013780 1565 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:56:09.018101 1565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4aa76475773049539f1aaa539293fd94: Generated new TSK 0
I20260619 08:56:09.018330 1565 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 08:56:09.052949 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38807
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:09.133121 1577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:09.133332 1577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:09.133353 1577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:09.134450 1577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:09.134485 1577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:09.134505 1577 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 08:56:09.134521 1577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 08:56:09.135962 1577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:09.136201 1577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:09.136397 1577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:09.141223 1585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:09.141299 1583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:09.141305 1582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:09.141455 1577 server_base.cc:1034] running on GCE node
I20260619 08:56:09.141640 1577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:09.141875 1577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:09.143010 1577 hybrid_clock.cc:648] HybridClock initialized: now 1781859369142997 us; error 30 us; skew 500 ppm
Jun 19 08:56:09 dist-test-slave-h4g5 krb5kdc[1460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:09.145437 1577 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 08:56:09.145851 1577 webserver.cc:460] Webserver started at http://127.0.98.1:33537/ using document root <none> and password file <none>
I20260619 08:56:09.146044 1577 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:09.146085 1577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:09.146205 1577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:09.147305 1577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-0/data/instance:
uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6"
format_stamp: "Formatted at 2026-06-19 08:56:09 on dist-test-slave-h4g5"
server_key: "54405317e6fd4d7a06d4f4c69dd83038"
server_key_iv: "dfff5d602ee4860ea82a57ee5ecf402e"
server_key_version: "encryptionkey@0"
I20260619 08:56:09.147643 1577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6"
format_stamp: "Formatted at 2026-06-19 08:56:09 on dist-test-slave-h4g5"
server_key: "54405317e6fd4d7a06d4f4c69dd83038"
server_key_iv: "dfff5d602ee4860ea82a57ee5ecf402e"
server_key_version: "encryptionkey@0"
I20260619 08:56:09.148986 1577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:56:09.149812 1592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:09.150007 1577 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:09.150098 1577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-0/wal
uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6"
format_stamp: "Formatted at 2026-06-19 08:56:09 on dist-test-slave-h4g5"
server_key: "54405317e6fd4d7a06d4f4c69dd83038"
server_key_iv: "dfff5d602ee4860ea82a57ee5ecf402e"
server_key_version: "encryptionkey@0"
I20260619 08:56:09.150164 1577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:09.161453 1577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:09.163357 1577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:09.163470 1577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:09.163967 1577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:09.164003 1577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:09.164029 1577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:09.164045 1577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 08:56:09 dist-test-slave-h4g5 krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:09.170864 1577 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45869
I20260619 08:56:09.171013 1707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45869 every 8 connection(s)
I20260619 08:56:09.171244 1577 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
I20260619 08:56:09.173329 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:09.164527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33843 (local address 127.0.98.62:38807)
0619 08:56:09.164592 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:09.164595 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:09.164798 (+ 203us) server_negotiation.cc:407] Connection header received
0619 08:56:09.165481 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:09.165483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:09.165496 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:09.165560 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:09.166453 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.166977 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.168017 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.168290 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.168348 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:09.170555 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:09.170556 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:09.170557 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:09.170564 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:09.172055 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:09.172468 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:09.172469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:09.172470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:09.172521 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:09.172775 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:09.172775 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:09.172776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:09.172905 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:09.172952 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:09.173169 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 08:56:09.173241 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 08:56:09.173498 1601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:09.164486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38807 (local address 127.0.98.1:33843)
0619 08:56:09.164651 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:09.164698 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:09.165359 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:09.165680 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:09.165685 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:09.166053 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:09.166350 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:09.166356 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.167065 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.167066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:09.167933 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:09.167936 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.168372 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.168373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:09.168481 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:09.168977 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:09.168993 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:09.170436 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:09.172172 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:09.172176 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:09.172178 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:09.172405 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:09.172602 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:09.172603 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:09.172603 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:09.172715 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:09.172954 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:09.172956 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:09.173071 (+ 115us) client_negotiation.cc:769] Sending connection context
0619 08:56:09.173148 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 08:56:09.173279 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":81,"threads_started":1}
I20260619 08:56:09.173964 1708 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38807
I20260619 08:56:09.174050 1708 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:09.174223 1708 heartbeater.cc:511] Master 127.0.98.62:38807 requested a full tablet report, sending...
I20260619 08:56:09.174747 1504 ts_manager.cc:194] Registered new tserver with Master: d1d8ba2d1e124bec8b100bfb4d2df3f6 (127.0.98.1:45869)
I20260619 08:56:09.175619 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33843
I20260619 08:56:09.177582 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 1577
I20260619 08:56:09.177675 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-0/wal/instance
I20260619 08:56:09.177845 392 external_mini_cluster.cc:1428] Setting key 7e6a793dccd767502cfedeecb7f21a12
I20260619 08:56:09.180441 1714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:09.174855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38807 (local address 127.0.98.1:53965)
0619 08:56:09.175037 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:09.175048 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:09.175106 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:09.175211 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:09.175211 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:09.175406 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:09.175471 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:09.175472 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.176309 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.176310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:09.176869 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:09.176871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.177183 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.177183 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:09.177223 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:09.177624 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:09.177628 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:09.177944 (+ 316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:09.179373 (+ 1429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:09.179374 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:09.179374 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:09.179574 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:09.179793 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:09.179794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:09.179794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:09.179846 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:09.180120 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:09.180121 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:09.180171 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 08:56:09.180281 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 08:56:09.180361 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":68,"threads_started":1}
I20260619 08:56:09.180473 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:09.174967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53965 (local address 127.0.98.62:38807)
0619 08:56:09.175004 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:09.175006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:09.175091 (+ 85us) server_negotiation.cc:407] Connection header received
0619 08:56:09.175156 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:09.175157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:09.175164 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:09.175207 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:09.175563 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.176222 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.176945 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.177126 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.177180 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:09.178037 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:09.178038 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:09.178039 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:09.178045 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:09.179297 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:09.179646 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:09.179648 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:09.179648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:09.179733 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:09.179894 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:09.179895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:09.179895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:09.180011 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:09.180071 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:09.180310 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 08:56:09.180386 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 08:56:09.202466 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38807
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:09.282851 1717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:09.283077 1717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:09.283107 1717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:09.284303 1717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:09.284348 1717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:09.284373 1717 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 08:56:09.284397 1717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 08:56:09.285910 1717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:09.286161 1717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:09.286382 1717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:09.291290 1722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:09.291241 1723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:09.291356 1725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:09.291553 1717 server_base.cc:1034] running on GCE node
I20260619 08:56:09.291767 1717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:09.291996 1717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:09.293159 1717 hybrid_clock.cc:648] HybridClock initialized: now 1781859369293145 us; error 30 us; skew 500 ppm
Jun 19 08:56:09 dist-test-slave-h4g5 krb5kdc[1460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:09.295719 1717 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 08:56:09.296202 1717 webserver.cc:460] Webserver started at http://127.0.98.2:34959/ using document root <none> and password file <none>
I20260619 08:56:09.296412 1717 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:09.296464 1717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:09.296590 1717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:09.297644 1717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-1/data/instance:
uuid: "0d8681ac3160473f9c09472f2b3ec4c5"
format_stamp: "Formatted at 2026-06-19 08:56:09 on dist-test-slave-h4g5"
server_key: "5fe24b5d7c4cacf1e7ac3ae82bf4630b"
server_key_iv: "7821e403d3e0af8156b6bf5f3fbec844"
server_key_version: "encryptionkey@0"
I20260619 08:56:09.297977 1717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0d8681ac3160473f9c09472f2b3ec4c5"
format_stamp: "Formatted at 2026-06-19 08:56:09 on dist-test-slave-h4g5"
server_key: "5fe24b5d7c4cacf1e7ac3ae82bf4630b"
server_key_iv: "7821e403d3e0af8156b6bf5f3fbec844"
server_key_version: "encryptionkey@0"
I20260619 08:56:09.299300 1717 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:56:09.300010 1732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:09.300169 1717 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:09.300252 1717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-1/wal
uuid: "0d8681ac3160473f9c09472f2b3ec4c5"
format_stamp: "Formatted at 2026-06-19 08:56:09 on dist-test-slave-h4g5"
server_key: "5fe24b5d7c4cacf1e7ac3ae82bf4630b"
server_key_iv: "7821e403d3e0af8156b6bf5f3fbec844"
server_key_version: "encryptionkey@0"
I20260619 08:56:09.300319 1717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:09.308583 1717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:09.310637 1717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:09.310777 1717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:09.311256 1717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:09.311293 1717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:09.311328 1717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:09.311347 1717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:09.318305 1717 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44941
I20260619 08:56:09.318778 1717 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 08:56:09 dist-test-slave-h4g5 krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:09.319542 1847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44941 every 8 connection(s)
I20260619 08:56:09.322654 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:09.311778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37667 (local address 127.0.98.62:38807)
0619 08:56:09.311830 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:09.311832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:09.312043 (+ 211us) server_negotiation.cc:407] Connection header received
0619 08:56:09.312804 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:09.312805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:09.312820 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:09.312915 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:09.313881 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.314645 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.316243 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.316644 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.316779 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:09.319832 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:09.319833 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:09.319834 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:09.319842 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:09.321261 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:09.321690 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:09.321692 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:09.321692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:09.321763 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:09.322035 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:09.322036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:09.322036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:09.322171 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:09.322227 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:09.322516 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 08:56:09.322587 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 08:56:09.322819 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:09.311778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38807 (local address 127.0.98.2:37667)
0619 08:56:09.311924 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:09.311958 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:09.312663 (+ 705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:09.313093 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:09.313098 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:09.313460 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:09.313759 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:09.313765 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.314766 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.314766 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:09.316117 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:09.316120 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.316834 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.316835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:09.317063 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:09.317899 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:09.317911 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:09.319741 (+ 1830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:09.321338 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:09.321345 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:09.321347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:09.321606 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:09.321829 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:09.321830 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:09.321831 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:09.321974 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:09.322224 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:09.322228 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:09.322395 (+ 167us) client_negotiation.cc:769] Sending connection context
0619 08:56:09.322488 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 08:56:09.322614 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":83,"threads_started":1}
I20260619 08:56:09.323285 1848 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38807
I20260619 08:56:09.323364 1848 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:09.323551 1848 heartbeater.cc:511] Master 127.0.98.62:38807 requested a full tablet report, sending...
I20260619 08:56:09.323958 1505 ts_manager.cc:194] Registered new tserver with Master: 0d8681ac3160473f9c09472f2b3ec4c5 (127.0.98.2:44941)
I20260619 08:56:09.324483 1505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37667
I20260619 08:56:09.327679 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 1717
I20260619 08:56:09.327792 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-1/wal/instance
I20260619 08:56:09.327955 392 external_mini_cluster.cc:1428] Setting key 75c86177566686dbcd8610c201de4921
I20260619 08:56:09.329840 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:09.324368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41629 (local address 127.0.98.62:38807)
0619 08:56:09.324615 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:09.324616 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:09.324623 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:56:09.324640 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:09.324641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:09.324650 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:09.324693 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:09.325105 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.325566 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.326181 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.326361 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.326401 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:09.327234 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:09.327235 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:09.327235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:09.327241 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:09.328755 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:09.329045 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:09.329046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:09.329046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:09.329100 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:09.329258 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:09.329258 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:09.329259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:09.329398 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:09.329455 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:09.329666 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 08:56:09.329747 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230}
I20260619 08:56:09.330143 1854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:09.324275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38807 (local address 127.0.98.2:41629)
0619 08:56:09.324402 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:09.324411 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:09.324452 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:09.324735 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:09.324736 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:09.324925 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:09.325019 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:09.325021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.325643 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.325644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:09.326125 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:09.326126 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.326406 (+ 280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.326407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:09.326480 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:09.326863 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:09.326866 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:09.327146 (+ 280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:09.328822 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:09.328823 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:09.328824 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:09.328986 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:09.329151 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:09.329152 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:09.329152 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:09.329209 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:09.329488 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:09.329489 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:09.329541 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 08:56:09.329928 (+ 387us) client_negotiation.cc:240] Negotiation successful
0619 08:56:09.330041 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":1,"thread_start_us":68,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 08:56:09.354423 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38807
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:09.433004 1857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:09.433212 1857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:09.433234 1857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:09.434361 1857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:09.434405 1857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:09.434432 1857 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 08:56:09.434451 1857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 08:56:09.435935 1857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:09.436177 1857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:09.436396 1857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:09.441325 1865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:09.441376 1863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:09.441362 1857 server_base.cc:1034] running on GCE node
W20260619 08:56:09.441331 1862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:09.441716 1857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:09.441991 1857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:09.443161 1857 hybrid_clock.cc:648] HybridClock initialized: now 1781859369443139 us; error 31 us; skew 500 ppm
Jun 19 08:56:09 dist-test-slave-h4g5 krb5kdc[1460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:09.445672 1857 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 08:56:09.446087 1857 webserver.cc:460] Webserver started at http://127.0.98.3:45525/ using document root <none> and password file <none>
I20260619 08:56:09.446310 1857 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:09.446354 1857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:09.446462 1857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:09.447540 1857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-2/data/instance:
uuid: "3d20be2a71344f39bc23a1eef0517831"
format_stamp: "Formatted at 2026-06-19 08:56:09 on dist-test-slave-h4g5"
server_key: "8b62276a60b11a4634127462e6b941a1"
server_key_iv: "bf79a5bf82e93e263b18c6f5fe11653b"
server_key_version: "encryptionkey@0"
I20260619 08:56:09.447860 1857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3d20be2a71344f39bc23a1eef0517831"
format_stamp: "Formatted at 2026-06-19 08:56:09 on dist-test-slave-h4g5"
server_key: "8b62276a60b11a4634127462e6b941a1"
server_key_iv: "bf79a5bf82e93e263b18c6f5fe11653b"
server_key_version: "encryptionkey@0"
I20260619 08:56:09.449295 1857 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:56:09.449994 1872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:09.450136 1857 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:09.450210 1857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-2/wal
uuid: "3d20be2a71344f39bc23a1eef0517831"
format_stamp: "Formatted at 2026-06-19 08:56:09 on dist-test-slave-h4g5"
server_key: "8b62276a60b11a4634127462e6b941a1"
server_key_iv: "bf79a5bf82e93e263b18c6f5fe11653b"
server_key_version: "encryptionkey@0"
I20260619 08:56:09.450284 1857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:09.464880 1857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:09.466832 1857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:09.466945 1857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:09.467469 1857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:09.467510 1857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:09.467532 1857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:09.467546 1857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:09.473596 1857 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33541
I20260619 08:56:09.473749 1987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33541 every 8 connection(s)
I20260619 08:56:09.474141 1857 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 08:56:09 dist-test-slave-h4g5 krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:09.477360 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:09.467963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55791 (local address 127.0.98.62:38807)
0619 08:56:09.468009 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:09.468011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:09.468262 (+ 251us) server_negotiation.cc:407] Connection header received
0619 08:56:09.468776 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:09.468777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:09.468789 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:09.468845 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:09.469703 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.470462 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.471930 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.472247 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.472327 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:09.474873 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:09.474874 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:09.474875 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:09.474882 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:09.476198 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:09.476543 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:09.476544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:09.476544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:09.476591 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:09.476826 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:09.476827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:09.476828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:09.476953 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:09.477022 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:09.477215 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 08:56:09.477291 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 08:56:09.477532 1881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:09.467969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38807 (local address 127.0.98.3:55791)
0619 08:56:09.468139 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:09.468186 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:09.468684 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:09.468974 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:09.468979 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:09.469316 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:09.469589 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:09.469595 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.470564 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.470565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:09.471841 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:09.471844 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.472328 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.472329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:09.472486 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:09.473064 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:09.473076 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:09.474781 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:09.476260 (+ 1479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:09.476265 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:09.476267 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:09.476480 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:09.476638 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:09.476639 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:09.476640 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:09.476744 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:09.477013 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:09.477016 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:09.477128 (+ 112us) client_negotiation.cc:769] Sending connection context
0619 08:56:09.477203 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 08:56:09.477322 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":75,"threads_started":1}
I20260619 08:56:09.477967 1988 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38807
I20260619 08:56:09.478049 1988 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:09.478222 1988 heartbeater.cc:511] Master 127.0.98.62:38807 requested a full tablet report, sending...
I20260619 08:56:09.478842 1504 ts_manager.cc:194] Registered new tserver with Master: 3d20be2a71344f39bc23a1eef0517831 (127.0.98.3:33541)
I20260619 08:56:09.479075 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 1857
I20260619 08:56:09.479162 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-2/wal/instance
I20260619 08:56:09.479311 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55791
I20260619 08:56:09.479329 392 external_mini_cluster.cc:1428] Setting key a1480d404a9b306c1e385e48cc936b8b
I20260619 08:56:09.480120 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:56:09.485091 1994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:09.478879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38807 (local address 127.0.98.3:59643)
0619 08:56:09.479098 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:09.479110 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:09.479175 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:09.479339 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:09.479340 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:09.479534 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:09.480122 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:09.480125 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.480758 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.480759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:09.481480 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:09.481482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.481904 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.481906 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:09.481979 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:09.482433 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:09.482438 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:09.482711 (+ 273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:09.484184 (+ 1473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:09.484186 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:09.484186 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:09.484350 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:09.484541 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:09.484542 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:09.484542 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:09.484603 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:09.484850 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:09.484851 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:09.484884 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 08:56:09.484930 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 08:56:09.485013 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":151,"threads_started":1}
I20260619 08:56:09.485113 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:09.478936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59643 (local address 127.0.98.62:38807)
0619 08:56:09.478977 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:09.478979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:09.479174 (+ 195us) server_negotiation.cc:407] Connection header received
0619 08:56:09.479220 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:09.479221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:09.479230 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:09.479282 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:09.480190 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.480667 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.481566 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.481752 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.481907 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:09.482791 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:09.482792 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:09.482793 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:09.482800 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:09.484119 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:09.484409 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:09.484410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:09.484410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:09.484467 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:09.484663 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:09.484664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:09.484664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:09.484792 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:09.484861 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:09.484957 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 08:56:09.485027 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Jun 19 08:56:09 dist-test-slave-h4g5 krb5kdc[1460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859369, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 08:56:09 dist-test-slave-h4g5 krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859369, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:09.510177 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:09.503393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36972 (local address 127.0.98.62:38807)
0619 08:56:09.503480 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:09.503482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:09.503491 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:56:09.503522 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:09.503523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:09.503532 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:09.503613 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:09.504271 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.504888 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.505646 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.505847 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.505903 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:09.507986 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:09.507987 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:09.507988 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:09.507996 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:09.509193 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:09.509503 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:09.509504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:09.509504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:09.509558 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:09.509712 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:09.509712 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:09.509713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:09.509827 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:09.509915 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:09.510004 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 08:56:09.510081 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 08:56:09.511737 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:56:09.513367 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:56:09.519536 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:09.515352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44941 (local address 127.0.0.1:39930)
0619 08:56:09.515827 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:09.515873 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:09.515949 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:09.516222 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:09.516225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:09.516227 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:09.516307 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:09.516309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.517204 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.517204 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:09.518303 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:09.518305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.519208 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.519209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:09.519284 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:09.519299 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 08:56:09.519369 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 08:56:09.519436 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":2,"thread_start_us":77,"threads_started":1}
I20260619 08:56:09.519552 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:09.515506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39930 (local address 127.0.98.2:44941)
0619 08:56:09.515665 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:09.515670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:09.515968 (+ 298us) server_negotiation.cc:407] Connection header received
0619 08:56:09.516034 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:09.516036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:09.516054 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:09.516146 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:09.516387 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.517126 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.518389 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.519122 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.519194 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:09.519281 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:09.519388 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 08:56:09.519430 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
I20260619 08:56:09.519670 2002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:09.515374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33541 (local address 127.0.0.1:40940)
0619 08:56:09.515855 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:09.515884 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:09.515951 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:09.516240 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:09.516241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:09.516243 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:09.516315 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:09.516317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.517641 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.517641 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:09.518606 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:09.518608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.519384 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.519384 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:09.519438 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:09.519440 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:56:09.519484 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 08:56:09.519517 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":31,"thread_start_us":60,"threads_started":1}
I20260619 08:56:09.519706 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:09.515692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40940 (local address 127.0.98.3:33541)
0619 08:56:09.515859 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:09.515863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:09.515937 (+ 74us) server_negotiation.cc:407] Connection header received
0619 08:56:09.516038 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:09.516040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:09.516055 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:09.516172 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:09.516390 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.517547 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.518677 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.519212 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.519278 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:09.519352 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:09.519573 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 08:56:09.519612 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":83,"threads_started":1}
I20260619 08:56:09.519838 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:09.515557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45869 (local address 127.0.0.1:33712)
0619 08:56:09.516040 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:09.516048 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:09.516147 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:09.516365 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:09.516366 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:09.516368 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:09.516405 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:09.516407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.517637 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.517639 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:09.518674 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:09.518677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.519596 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.519597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:09.519657 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:09.519662 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 08:56:09.519711 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 08:56:09.519768 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":4,"thread_start_us":38,"threads_started":1}
I20260619 08:56:09.520057 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:09.515665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33712 (local address 127.0.98.1:45869)
0619 08:56:09.515830 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:09.515833 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:09.516122 (+ 289us) server_negotiation.cc:407] Connection header received
0619 08:56:09.516228 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:09.516231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:09.516244 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:09.516306 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:09.516486 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.517546 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.518755 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.519428 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.519490 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:09.519561 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:09.519927 (+ 366us) server_negotiation.cc:299] Negotiation successful
0619 08:56:09.519965 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":77,"threads_started":1}
I20260619 08:56:09.520794 1922 tablet_service.cc:1467] Processing CreateTablet for tablet 96748c6945264653ae19ebd76803695b (DEFAULT_TABLE table=test-table [id=34667bec2fbb4f5cb65fab8130418cc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:09.520792 1642 tablet_service.cc:1467] Processing CreateTablet for tablet 96748c6945264653ae19ebd76803695b (DEFAULT_TABLE table=test-table [id=34667bec2fbb4f5cb65fab8130418cc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:09.520879 1781 tablet_service.cc:1467] Processing CreateTablet for tablet 96748c6945264653ae19ebd76803695b (DEFAULT_TABLE table=test-table [id=34667bec2fbb4f5cb65fab8130418cc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:09.521167 1922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96748c6945264653ae19ebd76803695b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:09.521167 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96748c6945264653ae19ebd76803695b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:09.521178 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96748c6945264653ae19ebd76803695b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:09.524169 2008 tablet_bootstrap.cc:492] T 96748c6945264653ae19ebd76803695b P 0d8681ac3160473f9c09472f2b3ec4c5: Bootstrap starting.
I20260619 08:56:09.524703 2009 tablet_bootstrap.cc:492] T 96748c6945264653ae19ebd76803695b P 3d20be2a71344f39bc23a1eef0517831: Bootstrap starting.
I20260619 08:56:09.525038 2008 tablet_bootstrap.cc:654] T 96748c6945264653ae19ebd76803695b P 0d8681ac3160473f9c09472f2b3ec4c5: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:09.525187 2010 tablet_bootstrap.cc:492] T 96748c6945264653ae19ebd76803695b P d1d8ba2d1e124bec8b100bfb4d2df3f6: Bootstrap starting.
I20260619 08:56:09.525337 2009 tablet_bootstrap.cc:654] T 96748c6945264653ae19ebd76803695b P 3d20be2a71344f39bc23a1eef0517831: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:09.525516 2008 log.cc:824] T 96748c6945264653ae19ebd76803695b P 0d8681ac3160473f9c09472f2b3ec4c5: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:09.525642 2009 log.cc:824] T 96748c6945264653ae19ebd76803695b P 3d20be2a71344f39bc23a1eef0517831: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:09.525810 2010 tablet_bootstrap.cc:654] T 96748c6945264653ae19ebd76803695b P d1d8ba2d1e124bec8b100bfb4d2df3f6: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:09.526120 2010 log.cc:824] T 96748c6945264653ae19ebd76803695b P d1d8ba2d1e124bec8b100bfb4d2df3f6: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:09.526362 2009 tablet_bootstrap.cc:492] T 96748c6945264653ae19ebd76803695b P 3d20be2a71344f39bc23a1eef0517831: No bootstrap required, opened a new log
I20260619 08:56:09.526362 2008 tablet_bootstrap.cc:492] T 96748c6945264653ae19ebd76803695b P 0d8681ac3160473f9c09472f2b3ec4c5: No bootstrap required, opened a new log
I20260619 08:56:09.526443 2009 ts_tablet_manager.cc:1397] T 96748c6945264653ae19ebd76803695b P 3d20be2a71344f39bc23a1eef0517831: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:56:09.526439 2008 ts_tablet_manager.cc:1397] T 96748c6945264653ae19ebd76803695b P 0d8681ac3160473f9c09472f2b3ec4c5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:56:09.526780 2010 tablet_bootstrap.cc:492] T 96748c6945264653ae19ebd76803695b P d1d8ba2d1e124bec8b100bfb4d2df3f6: No bootstrap required, opened a new log
I20260619 08:56:09.526844 2010 ts_tablet_manager.cc:1397] T 96748c6945264653ae19ebd76803695b P d1d8ba2d1e124bec8b100bfb4d2df3f6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:56:09.528426 2010 raft_consensus.cc:348] T 96748c6945264653ae19ebd76803695b P d1d8ba2d1e124bec8b100bfb4d2df3f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } } peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } }
I20260619 08:56:09.528568 2010 raft_consensus.cc:374] T 96748c6945264653ae19ebd76803695b P d1d8ba2d1e124bec8b100bfb4d2df3f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:09.528599 2010 raft_consensus.cc:729] T 96748c6945264653ae19ebd76803695b P d1d8ba2d1e124bec8b100bfb4d2df3f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1d8ba2d1e124bec8b100bfb4d2df3f6, State: Initialized, Role: FOLLOWER
I20260619 08:56:09.528479 2008 raft_consensus.cc:348] T 96748c6945264653ae19ebd76803695b P 0d8681ac3160473f9c09472f2b3ec4c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } } peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } }
I20260619 08:56:09.528479 2009 raft_consensus.cc:348] T 96748c6945264653ae19ebd76803695b P 3d20be2a71344f39bc23a1eef0517831 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } } peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } }
I20260619 08:56:09.528636 2008 raft_consensus.cc:374] T 96748c6945264653ae19ebd76803695b P 0d8681ac3160473f9c09472f2b3ec4c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:09.528636 2009 raft_consensus.cc:374] T 96748c6945264653ae19ebd76803695b P 3d20be2a71344f39bc23a1eef0517831 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:09.528669 2008 raft_consensus.cc:729] T 96748c6945264653ae19ebd76803695b P 0d8681ac3160473f9c09472f2b3ec4c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d8681ac3160473f9c09472f2b3ec4c5, State: Initialized, Role: FOLLOWER
I20260619 08:56:09.528669 2009 raft_consensus.cc:729] T 96748c6945264653ae19ebd76803695b P 3d20be2a71344f39bc23a1eef0517831 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d20be2a71344f39bc23a1eef0517831, State: Initialized, Role: FOLLOWER
I20260619 08:56:09.528739 2010 consensus_queue.cc:260] T 96748c6945264653ae19ebd76803695b P d1d8ba2d1e124bec8b100bfb4d2df3f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } } peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } }
I20260619 08:56:09.528774 2008 consensus_queue.cc:260] T 96748c6945264653ae19ebd76803695b P 0d8681ac3160473f9c09472f2b3ec4c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } } peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } }
I20260619 08:56:09.528774 2009 consensus_queue.cc:260] T 96748c6945264653ae19ebd76803695b P 3d20be2a71344f39bc23a1eef0517831 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } } peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } }
I20260619 08:56:09.529011 2010 ts_tablet_manager.cc:1428] T 96748c6945264653ae19ebd76803695b P d1d8ba2d1e124bec8b100bfb4d2df3f6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:56:09.529022 2009 ts_tablet_manager.cc:1428] T 96748c6945264653ae19ebd76803695b P 3d20be2a71344f39bc23a1eef0517831: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:56:09.529022 2008 ts_tablet_manager.cc:1428] T 96748c6945264653ae19ebd76803695b P 0d8681ac3160473f9c09472f2b3ec4c5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:56:09.529124 1708 heartbeater.cc:503] Master 127.0.98.62:38807 was elected leader, sending a full tablet report...
I20260619 08:56:09.529134 1988 heartbeater.cc:503] Master 127.0.98.62:38807 was elected leader, sending a full tablet report...
I20260619 08:56:09.529165 1848 heartbeater.cc:503] Master 127.0.98.62:38807 was elected leader, sending a full tablet report...
I20260619 08:56:09.545074 2014 raft_consensus.cc:482] T 96748c6945264653ae19ebd76803695b P d1d8ba2d1e124bec8b100bfb4d2df3f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:56:09.545199 2014 raft_consensus.cc:504] T 96748c6945264653ae19ebd76803695b P d1d8ba2d1e124bec8b100bfb4d2df3f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } } peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } }
I20260619 08:56:09.545528 2014 leader_election.cc:288] T 96748c6945264653ae19ebd76803695b P d1d8ba2d1e124bec8b100bfb4d2df3f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d8681ac3160473f9c09472f2b3ec4c5 (127.0.98.2:44941), 3d20be2a71344f39bc23a1eef0517831 (127.0.98.3:33541)
I20260619 08:56:09.548460 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:09.545793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49441 (local address 127.0.98.2:44941)
0619 08:56:09.545833 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:09.545834 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:09.545968 (+ 134us) server_negotiation.cc:407] Connection header received
0619 08:56:09.546002 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:09.546003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:09.546015 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:09.546061 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:09.546190 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.546655 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.547711 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.548154 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.548220 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:09.548231 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:09.548357 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 08:56:09.548392 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 08:56:09.548497 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:09.545775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57313 (local address 127.0.98.3:33541)
0619 08:56:09.545820 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:09.545822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:09.545833 (+ 11us) server_negotiation.cc:407] Connection header received
0619 08:56:09.545906 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:09.545907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:09.545917 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:09.545976 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:09.546115 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.546651 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.547695 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.548181 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.548238 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:09.548248 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:09.548386 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 08:56:09.548421 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 08:56:09.548506 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:09.545705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44941 (local address 127.0.98.1:49441)
0619 08:56:09.545902 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:09.545910 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:09.545962 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:09.546073 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:09.546074 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:09.546075 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:09.546138 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:09.546140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.546733 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.546734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:09.547618 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:09.547620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.548219 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.548220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:09.548290 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:09.548293 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:56:09.548338 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 08:56:09.548377 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":55,"thread_start_us":112,"threads_started":1}
I20260619 08:56:09.548521 1601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:09.545704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33541 (local address 127.0.98.1:57313)
0619 08:56:09.545762 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:09.545773 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:09.545818 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:09.545990 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:09.545991 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:09.545993 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:09.546058 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:09.546061 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.546729 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.546730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:09.547627 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:09.547630 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.548256 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.548257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:09.548332 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:09.548334 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:56:09.548377 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 08:56:09.548421 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":42}
I20260619 08:56:09.548856 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96748c6945264653ae19ebd76803695b" candidate_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" is_pre_election: true
I20260619 08:56:09.548841 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96748c6945264653ae19ebd76803695b" candidate_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d20be2a71344f39bc23a1eef0517831" is_pre_election: true
I20260619 08:56:09.549008 1801 raft_consensus.cc:2454] T 96748c6945264653ae19ebd76803695b P 0d8681ac3160473f9c09472f2b3ec4c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1d8ba2d1e124bec8b100bfb4d2df3f6 in term 0.
I20260619 08:56:09.549008 1942 raft_consensus.cc:2454] T 96748c6945264653ae19ebd76803695b P 3d20be2a71344f39bc23a1eef0517831 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1d8ba2d1e124bec8b100bfb4d2df3f6 in term 0.
I20260619 08:56:09.549222 1593 leader_election.cc:302] T 96748c6945264653ae19ebd76803695b P d1d8ba2d1e124bec8b100bfb4d2df3f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d8681ac3160473f9c09472f2b3ec4c5, d1d8ba2d1e124bec8b100bfb4d2df3f6; no voters:
I20260619 08:56:09.549347 2014 raft_consensus.cc:2781] T 96748c6945264653ae19ebd76803695b P d1d8ba2d1e124bec8b100bfb4d2df3f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:56:09.549403 2014 raft_consensus.cc:482] T 96748c6945264653ae19ebd76803695b P d1d8ba2d1e124bec8b100bfb4d2df3f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:56:09.549434 2014 raft_consensus.cc:3037] T 96748c6945264653ae19ebd76803695b P d1d8ba2d1e124bec8b100bfb4d2df3f6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:09.550334 2014 raft_consensus.cc:504] T 96748c6945264653ae19ebd76803695b P d1d8ba2d1e124bec8b100bfb4d2df3f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } } peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } }
I20260619 08:56:09.550473 2014 leader_election.cc:288] T 96748c6945264653ae19ebd76803695b P d1d8ba2d1e124bec8b100bfb4d2df3f6 [CANDIDATE]: Term 1 election: Requested vote from peers 0d8681ac3160473f9c09472f2b3ec4c5 (127.0.98.2:44941), 3d20be2a71344f39bc23a1eef0517831 (127.0.98.3:33541)
I20260619 08:56:09.550627 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96748c6945264653ae19ebd76803695b" candidate_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d20be2a71344f39bc23a1eef0517831"
I20260619 08:56:09.550675 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96748c6945264653ae19ebd76803695b" candidate_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d8681ac3160473f9c09472f2b3ec4c5"
I20260619 08:56:09.550714 1942 raft_consensus.cc:3037] T 96748c6945264653ae19ebd76803695b P 3d20be2a71344f39bc23a1eef0517831 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:09.550745 1801 raft_consensus.cc:3037] T 96748c6945264653ae19ebd76803695b P 0d8681ac3160473f9c09472f2b3ec4c5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:09.551436 1942 raft_consensus.cc:2454] T 96748c6945264653ae19ebd76803695b P 3d20be2a71344f39bc23a1eef0517831 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1d8ba2d1e124bec8b100bfb4d2df3f6 in term 1.
I20260619 08:56:09.551436 1801 raft_consensus.cc:2454] T 96748c6945264653ae19ebd76803695b P 0d8681ac3160473f9c09472f2b3ec4c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1d8ba2d1e124bec8b100bfb4d2df3f6 in term 1.
I20260619 08:56:09.551584 1593 leader_election.cc:302] T 96748c6945264653ae19ebd76803695b P d1d8ba2d1e124bec8b100bfb4d2df3f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d8681ac3160473f9c09472f2b3ec4c5, d1d8ba2d1e124bec8b100bfb4d2df3f6; no voters:
I20260619 08:56:09.551702 2014 raft_consensus.cc:2781] T 96748c6945264653ae19ebd76803695b P d1d8ba2d1e124bec8b100bfb4d2df3f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:09.551862 2014 raft_consensus.cc:686] T 96748c6945264653ae19ebd76803695b P d1d8ba2d1e124bec8b100bfb4d2df3f6 [term 1 LEADER]: Becoming Leader. State: Replica: d1d8ba2d1e124bec8b100bfb4d2df3f6, State: Running, Role: LEADER
I20260619 08:56:09.551951 2014 consensus_queue.cc:237] T 96748c6945264653ae19ebd76803695b P d1d8ba2d1e124bec8b100bfb4d2df3f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } } peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } }
I20260619 08:56:09.552824 1505 catalog_manager.cc:5495] T 96748c6945264653ae19ebd76803695b P d1d8ba2d1e124bec8b100bfb4d2df3f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1d8ba2d1e124bec8b100bfb4d2df3f6 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } health_report { overall_health: HEALTHY } } }
I20260619 08:56:09.560525 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:09.558166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33722 (local address 127.0.98.1:45869)
0619 08:56:09.558217 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:09.558219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:09.558226 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:56:09.558256 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:09.558257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:09.558274 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:09.558328 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:56:09.558464 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.558923 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.559667 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.559870 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.559924 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:09.560146 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:56:09.560250 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:56:09.560321 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:09.560417 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 08:56:09.560446 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 08:56:09.562297 1942 raft_consensus.cc:1261] T 96748c6945264653ae19ebd76803695b P 3d20be2a71344f39bc23a1eef0517831 [term 1 FOLLOWER]: Refusing update from remote peer d1d8ba2d1e124bec8b100bfb4d2df3f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:56:09.562312 1801 raft_consensus.cc:1261] T 96748c6945264653ae19ebd76803695b P 0d8681ac3160473f9c09472f2b3ec4c5 [term 1 FOLLOWER]: Refusing update from remote peer d1d8ba2d1e124bec8b100bfb4d2df3f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:56:09.562628 2018 consensus_queue.cc:1035] T 96748c6945264653ae19ebd76803695b P d1d8ba2d1e124bec8b100bfb4d2df3f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:56:09.562759 2014 consensus_queue.cc:1035] T 96748c6945264653ae19ebd76803695b P d1d8ba2d1e124bec8b100bfb4d2df3f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:56:09.565093 2021 mvcc.cc:205] Tried to move back new op lower bound from 7298495977724420096 to 7298495977685286912. Current Snapshot: MvccSnapshot[applied={T|T < 7298495977724420096}]
I20260619 08:56:09.565271 2022 mvcc.cc:205] Tried to move back new op lower bound from 7298495977724420096 to 7298495977685286912. Current Snapshot: MvccSnapshot[applied={T|T < 7298495977724420096}]
I20260619 08:56:09.566275 2023 mvcc.cc:205] Tried to move back new op lower bound from 7298495977724420096 to 7298495977685286912. Current Snapshot: MvccSnapshot[applied={T|T < 7298495977724420096}]
I20260619 08:56:09.569233 1504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:56:09.569343 1504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:56:09.570369 1504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4aa76475773049539f1aaa539293fd94: Sending DeleteTablet for 3 replicas of tablet 96748c6945264653ae19ebd76803695b
I20260619 08:56:09.570729 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet 96748c6945264653ae19ebd76803695b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39930
I20260619 08:56:09.570755 1922 tablet_service.cc:1514] Processing DeleteTablet for tablet 96748c6945264653ae19ebd76803695b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40940
I20260619 08:56:09.571031 1642 tablet_service.cc:1514] Processing DeleteTablet for tablet 96748c6945264653ae19ebd76803695b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33712
I20260619 08:56:09.571182 2031 tablet_replica.cc:330] T 96748c6945264653ae19ebd76803695b P 3d20be2a71344f39bc23a1eef0517831: stopping tablet replica
I20260619 08:56:09.571144 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:56:09.571276 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:56:09.571312 2033 tablet_replica.cc:330] T 96748c6945264653ae19ebd76803695b P d1d8ba2d1e124bec8b100bfb4d2df3f6: stopping tablet replica
I20260619 08:56:09.571499 2031 raft_consensus.cc:2229] T 96748c6945264653ae19ebd76803695b P 3d20be2a71344f39bc23a1eef0517831 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:56:09.571640 2031 raft_consensus.cc:2258] T 96748c6945264653ae19ebd76803695b P 3d20be2a71344f39bc23a1eef0517831 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:09.572043 2032 tablet_replica.cc:330] T 96748c6945264653ae19ebd76803695b P 0d8681ac3160473f9c09472f2b3ec4c5: stopping tablet replica
I20260619 08:56:09.572186 2032 raft_consensus.cc:2229] T 96748c6945264653ae19ebd76803695b P 0d8681ac3160473f9c09472f2b3ec4c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:56:09.572281 2031 ts_tablet_manager.cc:1905] T 96748c6945264653ae19ebd76803695b P 3d20be2a71344f39bc23a1eef0517831: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:09.572300 2032 raft_consensus.cc:2258] T 96748c6945264653ae19ebd76803695b P 0d8681ac3160473f9c09472f2b3ec4c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:09.572944 1922 tablet_service.cc:1467] Processing CreateTablet for tablet 7b96429d9e434ec1bff8ca110e3ef1fa (DEFAULT_TABLE table=test-table [id=645c3ee4a21a4d2b8f87d8bb04676f12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:09.572947 1781 tablet_service.cc:1467] Processing CreateTablet for tablet 7b96429d9e434ec1bff8ca110e3ef1fa (DEFAULT_TABLE table=test-table [id=645c3ee4a21a4d2b8f87d8bb04676f12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:09.573063 1922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b96429d9e434ec1bff8ca110e3ef1fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:09.573071 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b96429d9e434ec1bff8ca110e3ef1fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:09.573087 2032 ts_tablet_manager.cc:1905] T 96748c6945264653ae19ebd76803695b P 0d8681ac3160473f9c09472f2b3ec4c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:09.574326 2009 tablet_bootstrap.cc:492] T 7b96429d9e434ec1bff8ca110e3ef1fa P 3d20be2a71344f39bc23a1eef0517831: Bootstrap starting.
I20260619 08:56:09.574662 2008 tablet_bootstrap.cc:492] T 7b96429d9e434ec1bff8ca110e3ef1fa P 0d8681ac3160473f9c09472f2b3ec4c5: Bootstrap starting.
I20260619 08:56:09.574777 2009 tablet_bootstrap.cc:654] T 7b96429d9e434ec1bff8ca110e3ef1fa P 3d20be2a71344f39bc23a1eef0517831: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:09.575057 2008 tablet_bootstrap.cc:654] T 7b96429d9e434ec1bff8ca110e3ef1fa P 0d8681ac3160473f9c09472f2b3ec4c5: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:09.575613 1642 tablet_service.cc:1467] Processing CreateTablet for tablet 7b96429d9e434ec1bff8ca110e3ef1fa (DEFAULT_TABLE table=test-table [id=645c3ee4a21a4d2b8f87d8bb04676f12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:09.575764 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b96429d9e434ec1bff8ca110e3ef1fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:09.575922 2032 ts_tablet_manager.cc:1918] T 96748c6945264653ae19ebd76803695b P 0d8681ac3160473f9c09472f2b3ec4c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:56:09.575996 2032 log.cc:1196] T 96748c6945264653ae19ebd76803695b P 0d8681ac3160473f9c09472f2b3ec4c5: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-1/wal/wals/96748c6945264653ae19ebd76803695b
I20260619 08:56:09.576186 2031 ts_tablet_manager.cc:1918] T 96748c6945264653ae19ebd76803695b P 3d20be2a71344f39bc23a1eef0517831: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:56:09.576279 2032 ts_tablet_manager.cc:1939] T 96748c6945264653ae19ebd76803695b P 0d8681ac3160473f9c09472f2b3ec4c5: Deleting consensus metadata
I20260619 08:56:09.576354 2031 log.cc:1196] T 96748c6945264653ae19ebd76803695b P 3d20be2a71344f39bc23a1eef0517831: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-2/wal/wals/96748c6945264653ae19ebd76803695b
I20260619 08:56:09.576717 2031 ts_tablet_manager.cc:1939] T 96748c6945264653ae19ebd76803695b P 3d20be2a71344f39bc23a1eef0517831: Deleting consensus metadata
I20260619 08:56:09.576776 1489 catalog_manager.cc:4841] TS 0d8681ac3160473f9c09472f2b3ec4c5 (127.0.98.2:44941): tablet 96748c6945264653ae19ebd76803695b (table test-table [id=34667bec2fbb4f5cb65fab8130418cc5]) successfully deleted
I20260619 08:56:09.577309 1491 catalog_manager.cc:4841] TS 3d20be2a71344f39bc23a1eef0517831 (127.0.98.3:33541): tablet 96748c6945264653ae19ebd76803695b (table test-table [id=34667bec2fbb4f5cb65fab8130418cc5]) successfully deleted
I20260619 08:56:09.577354 2010 tablet_bootstrap.cc:492] T 7b96429d9e434ec1bff8ca110e3ef1fa P d1d8ba2d1e124bec8b100bfb4d2df3f6: Bootstrap starting.
I20260619 08:56:09.577452 2033 raft_consensus.cc:2229] T 96748c6945264653ae19ebd76803695b P d1d8ba2d1e124bec8b100bfb4d2df3f6 [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:56:09.577639 2033 raft_consensus.cc:2258] T 96748c6945264653ae19ebd76803695b P d1d8ba2d1e124bec8b100bfb4d2df3f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:09.577700 2008 tablet_bootstrap.cc:492] T 7b96429d9e434ec1bff8ca110e3ef1fa P 0d8681ac3160473f9c09472f2b3ec4c5: No bootstrap required, opened a new log
I20260619 08:56:09.577741 2010 tablet_bootstrap.cc:654] T 7b96429d9e434ec1bff8ca110e3ef1fa P d1d8ba2d1e124bec8b100bfb4d2df3f6: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:09.577757 2008 ts_tablet_manager.cc:1397] T 7b96429d9e434ec1bff8ca110e3ef1fa P 0d8681ac3160473f9c09472f2b3ec4c5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 08:56:09.577917 2008 raft_consensus.cc:348] T 7b96429d9e434ec1bff8ca110e3ef1fa P 0d8681ac3160473f9c09472f2b3ec4c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } }
I20260619 08:56:09.578006 2008 raft_consensus.cc:374] T 7b96429d9e434ec1bff8ca110e3ef1fa P 0d8681ac3160473f9c09472f2b3ec4c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:09.578033 2008 raft_consensus.cc:729] T 7b96429d9e434ec1bff8ca110e3ef1fa P 0d8681ac3160473f9c09472f2b3ec4c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d8681ac3160473f9c09472f2b3ec4c5, State: Initialized, Role: FOLLOWER
I20260619 08:56:09.578084 2008 consensus_queue.cc:260] T 7b96429d9e434ec1bff8ca110e3ef1fa P 0d8681ac3160473f9c09472f2b3ec4c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } }
I20260619 08:56:09.578097 2033 ts_tablet_manager.cc:1905] T 96748c6945264653ae19ebd76803695b P d1d8ba2d1e124bec8b100bfb4d2df3f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:09.578183 2008 ts_tablet_manager.cc:1428] T 7b96429d9e434ec1bff8ca110e3ef1fa P 0d8681ac3160473f9c09472f2b3ec4c5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:09.578414 2009 tablet_bootstrap.cc:492] T 7b96429d9e434ec1bff8ca110e3ef1fa P 3d20be2a71344f39bc23a1eef0517831: No bootstrap required, opened a new log
I20260619 08:56:09.578470 2009 ts_tablet_manager.cc:1397] T 7b96429d9e434ec1bff8ca110e3ef1fa P 3d20be2a71344f39bc23a1eef0517831: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 08:56:09.578598 2009 raft_consensus.cc:348] T 7b96429d9e434ec1bff8ca110e3ef1fa P 3d20be2a71344f39bc23a1eef0517831 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } }
I20260619 08:56:09.578655 2009 raft_consensus.cc:374] T 7b96429d9e434ec1bff8ca110e3ef1fa P 3d20be2a71344f39bc23a1eef0517831 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:09.578675 2009 raft_consensus.cc:729] T 7b96429d9e434ec1bff8ca110e3ef1fa P 3d20be2a71344f39bc23a1eef0517831 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d20be2a71344f39bc23a1eef0517831, State: Initialized, Role: FOLLOWER
I20260619 08:56:09.578712 2009 consensus_queue.cc:260] T 7b96429d9e434ec1bff8ca110e3ef1fa P 3d20be2a71344f39bc23a1eef0517831 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } }
I20260619 08:56:09.578811 2009 ts_tablet_manager.cc:1428] T 7b96429d9e434ec1bff8ca110e3ef1fa P 3d20be2a71344f39bc23a1eef0517831: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:09.579730 2033 ts_tablet_manager.cc:1918] T 96748c6945264653ae19ebd76803695b P d1d8ba2d1e124bec8b100bfb4d2df3f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:56:09.579841 2033 log.cc:1196] T 96748c6945264653ae19ebd76803695b P d1d8ba2d1e124bec8b100bfb4d2df3f6: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-0/wal/wals/96748c6945264653ae19ebd76803695b
I20260619 08:56:09.580132 2033 ts_tablet_manager.cc:1939] T 96748c6945264653ae19ebd76803695b P d1d8ba2d1e124bec8b100bfb4d2df3f6: Deleting consensus metadata
I20260619 08:56:09.580511 2010 tablet_bootstrap.cc:492] T 7b96429d9e434ec1bff8ca110e3ef1fa P d1d8ba2d1e124bec8b100bfb4d2df3f6: No bootstrap required, opened a new log
I20260619 08:56:09.580559 2010 ts_tablet_manager.cc:1397] T 7b96429d9e434ec1bff8ca110e3ef1fa P d1d8ba2d1e124bec8b100bfb4d2df3f6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 08:56:09.580559 1491 catalog_manager.cc:4841] TS d1d8ba2d1e124bec8b100bfb4d2df3f6 (127.0.98.1:45869): tablet 96748c6945264653ae19ebd76803695b (table test-table [id=34667bec2fbb4f5cb65fab8130418cc5]) successfully deleted
I20260619 08:56:09.580693 2010 raft_consensus.cc:348] T 7b96429d9e434ec1bff8ca110e3ef1fa P d1d8ba2d1e124bec8b100bfb4d2df3f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } }
I20260619 08:56:09.580749 2010 raft_consensus.cc:374] T 7b96429d9e434ec1bff8ca110e3ef1fa P d1d8ba2d1e124bec8b100bfb4d2df3f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:09.580766 2010 raft_consensus.cc:729] T 7b96429d9e434ec1bff8ca110e3ef1fa P d1d8ba2d1e124bec8b100bfb4d2df3f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1d8ba2d1e124bec8b100bfb4d2df3f6, State: Initialized, Role: FOLLOWER
I20260619 08:56:09.580806 2010 consensus_queue.cc:260] T 7b96429d9e434ec1bff8ca110e3ef1fa P d1d8ba2d1e124bec8b100bfb4d2df3f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } }
I20260619 08:56:09.580886 2010 ts_tablet_manager.cc:1428] T 7b96429d9e434ec1bff8ca110e3ef1fa P d1d8ba2d1e124bec8b100bfb4d2df3f6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260619 08:56:09.672472 1709 tablet.cc:2369] T 7b96429d9e434ec1bff8ca110e3ef1fa P d1d8ba2d1e124bec8b100bfb4d2df3f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 08:56:09.725124 1989 tablet.cc:2369] T 7b96429d9e434ec1bff8ca110e3ef1fa P 3d20be2a71344f39bc23a1eef0517831: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:56:09.797827 2015 raft_consensus.cc:482] T 7b96429d9e434ec1bff8ca110e3ef1fa P 0d8681ac3160473f9c09472f2b3ec4c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:56:09.797935 2015 raft_consensus.cc:504] T 7b96429d9e434ec1bff8ca110e3ef1fa P 0d8681ac3160473f9c09472f2b3ec4c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } }
I20260619 08:56:09.798246 2015 leader_election.cc:288] T 7b96429d9e434ec1bff8ca110e3ef1fa P 0d8681ac3160473f9c09472f2b3ec4c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1d8ba2d1e124bec8b100bfb4d2df3f6 (127.0.98.1:45869), 3d20be2a71344f39bc23a1eef0517831 (127.0.98.3:33541)
I20260619 08:56:09.802052 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:09.798504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33661 (local address 127.0.98.1:45869)
0619 08:56:09.798572 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:09.798574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:09.798584 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:56:09.798615 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:09.798616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:09.798625 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:09.798684 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:09.798848 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.799598 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.801147 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.801684 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.801758 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:09.801767 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:09.801938 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 08:56:09.801982 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 08:56:09.802052 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:09.798631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39437 (local address 127.0.98.3:33541)
0619 08:56:09.798682 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:09.798683 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:09.798692 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:56:09.798755 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:09.798756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:09.798762 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:09.798801 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:09.798972 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.799650 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.801138 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.801697 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.801758 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:09.801767 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:09.801938 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 08:56:09.801982 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 08:56:09.802057 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:09.798409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45869 (local address 127.0.98.2:33661)
0619 08:56:09.798464 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:09.798479 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:09.798540 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:09.798697 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:09.798698 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:09.798700 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:09.798784 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:09.798785 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.799716 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.799717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:09.801057 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:09.801060 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.801766 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.801767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:09.801848 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:09.801849 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 08:56:09.801905 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 08:56:09.801951 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":35}
I20260619 08:56:09.802075 2037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:09.798539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33541 (local address 127.0.98.2:39437)
0619 08:56:09.798644 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:09.798654 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:09.798702 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:09.798819 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:09.798820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:09.798821 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:09.798911 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:09.798912 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.799726 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.799726 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:09.801057 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:09.801060 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.801778 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.801779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:09.801844 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:09.801849 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 08:56:09.801905 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 08:56:09.801951 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":81,"thread_start_us":64,"threads_started":1}
I20260619 08:56:09.802361 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b96429d9e434ec1bff8ca110e3ef1fa" candidate_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d20be2a71344f39bc23a1eef0517831" is_pre_election: true
I20260619 08:56:09.802450 1942 raft_consensus.cc:2454] T 7b96429d9e434ec1bff8ca110e3ef1fa P 3d20be2a71344f39bc23a1eef0517831 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d8681ac3160473f9c09472f2b3ec4c5 in term 0.
I20260619 08:56:09.802388 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b96429d9e434ec1bff8ca110e3ef1fa" candidate_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" is_pre_election: true
I20260619 08:56:09.802513 1662 raft_consensus.cc:2454] T 7b96429d9e434ec1bff8ca110e3ef1fa P d1d8ba2d1e124bec8b100bfb4d2df3f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d8681ac3160473f9c09472f2b3ec4c5 in term 0.
I20260619 08:56:09.802614 1735 leader_election.cc:302] T 7b96429d9e434ec1bff8ca110e3ef1fa P 0d8681ac3160473f9c09472f2b3ec4c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d8681ac3160473f9c09472f2b3ec4c5, 3d20be2a71344f39bc23a1eef0517831; no voters:
I20260619 08:56:09.802771 2015 raft_consensus.cc:2781] T 7b96429d9e434ec1bff8ca110e3ef1fa P 0d8681ac3160473f9c09472f2b3ec4c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:56:09.802821 2015 raft_consensus.cc:482] T 7b96429d9e434ec1bff8ca110e3ef1fa P 0d8681ac3160473f9c09472f2b3ec4c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:56:09.802842 2015 raft_consensus.cc:3037] T 7b96429d9e434ec1bff8ca110e3ef1fa P 0d8681ac3160473f9c09472f2b3ec4c5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:09.803598 2015 raft_consensus.cc:504] T 7b96429d9e434ec1bff8ca110e3ef1fa P 0d8681ac3160473f9c09472f2b3ec4c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } }
I20260619 08:56:09.803705 2015 leader_election.cc:288] T 7b96429d9e434ec1bff8ca110e3ef1fa P 0d8681ac3160473f9c09472f2b3ec4c5 [CANDIDATE]: Term 1 election: Requested vote from peers d1d8ba2d1e124bec8b100bfb4d2df3f6 (127.0.98.1:45869), 3d20be2a71344f39bc23a1eef0517831 (127.0.98.3:33541)
I20260619 08:56:09.803882 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b96429d9e434ec1bff8ca110e3ef1fa" candidate_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6"
I20260619 08:56:09.803917 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b96429d9e434ec1bff8ca110e3ef1fa" candidate_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d20be2a71344f39bc23a1eef0517831"
I20260619 08:56:09.803957 1662 raft_consensus.cc:3037] T 7b96429d9e434ec1bff8ca110e3ef1fa P d1d8ba2d1e124bec8b100bfb4d2df3f6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:09.803979 1942 raft_consensus.cc:3037] T 7b96429d9e434ec1bff8ca110e3ef1fa P 3d20be2a71344f39bc23a1eef0517831 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:09.804697 1662 raft_consensus.cc:2454] T 7b96429d9e434ec1bff8ca110e3ef1fa P d1d8ba2d1e124bec8b100bfb4d2df3f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d8681ac3160473f9c09472f2b3ec4c5 in term 1.
I20260619 08:56:09.804697 1942 raft_consensus.cc:2454] T 7b96429d9e434ec1bff8ca110e3ef1fa P 3d20be2a71344f39bc23a1eef0517831 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d8681ac3160473f9c09472f2b3ec4c5 in term 1.
I20260619 08:56:09.804880 1735 leader_election.cc:302] T 7b96429d9e434ec1bff8ca110e3ef1fa P 0d8681ac3160473f9c09472f2b3ec4c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d8681ac3160473f9c09472f2b3ec4c5, d1d8ba2d1e124bec8b100bfb4d2df3f6; no voters:
I20260619 08:56:09.804960 2015 raft_consensus.cc:2781] T 7b96429d9e434ec1bff8ca110e3ef1fa P 0d8681ac3160473f9c09472f2b3ec4c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:09.805061 2015 raft_consensus.cc:686] T 7b96429d9e434ec1bff8ca110e3ef1fa P 0d8681ac3160473f9c09472f2b3ec4c5 [term 1 LEADER]: Becoming Leader. State: Replica: 0d8681ac3160473f9c09472f2b3ec4c5, State: Running, Role: LEADER
I20260619 08:56:09.805125 2015 consensus_queue.cc:237] T 7b96429d9e434ec1bff8ca110e3ef1fa P 0d8681ac3160473f9c09472f2b3ec4c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } }
I20260619 08:56:09.805951 1505 catalog_manager.cc:5495] T 7b96429d9e434ec1bff8ca110e3ef1fa P 0d8681ac3160473f9c09472f2b3ec4c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d8681ac3160473f9c09472f2b3ec4c5 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } health_report { overall_health: UNKNOWN } } }
W20260619 08:56:09.820487 1849 tablet.cc:2369] T 7b96429d9e434ec1bff8ca110e3ef1fa P 0d8681ac3160473f9c09472f2b3ec4c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 08:56:09 dist-test-slave-h4g5 krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:09.879786 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:09.872780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38807 (local address 127.0.0.1:36976)
0619 08:56:09.872985 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:09.872996 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:09.873076 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:09.873189 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:09.873189 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:09.873504 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:09.873550 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:09.873552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.874240 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.874241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:09.874822 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:09.874825 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.875246 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.875247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:09.875289 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:09.875753 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:09.875757 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:09.877074 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:09.878600 (+ 1526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:09.878607 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:09.878609 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:09.878803 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:09.878987 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:09.878988 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:09.878988 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:09.879048 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:09.879366 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:09.879374 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:09.879450 (+ 76us) client_negotiation.cc:769] Sending connection context
0619 08:56:09.879538 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 08:56:09.879668 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":96,"threads_started":1}
I20260619 08:56:09.879884 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:09.872866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36976 (local address 127.0.98.62:38807)
0619 08:56:09.872958 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:09.872961 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:09.873066 (+ 105us) server_negotiation.cc:407] Connection header received
0619 08:56:09.873136 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:09.873137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:09.873145 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:09.873186 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:09.873627 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.874168 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.874969 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:09.875176 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:09.875241 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:09.877153 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:09.877154 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:09.877154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:09.877160 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:09.878519 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:09.878869 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:09.878870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:09.878871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:09.878925 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:09.879129 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:09.879131 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:09.879131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:09.879265 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:09.879340 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:09.879569 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 08:56:09.879759 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":62}
I20260619 08:56:09.881867 1505 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36976:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 08:56:09.882076 1505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:56:09.883469 1781 tablet_service.cc:1467] Processing CreateTablet for tablet 1aa5581fb8b34901afeac1ca53f74736 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1f8eef78b13e4f2783fdd769a602e202]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 08:56:09.883455 1642 tablet_service.cc:1467] Processing CreateTablet for tablet 1aa5581fb8b34901afeac1ca53f74736 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1f8eef78b13e4f2783fdd769a602e202]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 08:56:09.883616 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aa5581fb8b34901afeac1ca53f74736. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:09.883625 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aa5581fb8b34901afeac1ca53f74736. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:09.884830 2010 tablet_bootstrap.cc:492] T 1aa5581fb8b34901afeac1ca53f74736 P d1d8ba2d1e124bec8b100bfb4d2df3f6: Bootstrap starting.
I20260619 08:56:09.884855 2008 tablet_bootstrap.cc:492] T 1aa5581fb8b34901afeac1ca53f74736 P 0d8681ac3160473f9c09472f2b3ec4c5: Bootstrap starting.
I20260619 08:56:09.884922 1922 tablet_service.cc:1467] Processing CreateTablet for tablet 1aa5581fb8b34901afeac1ca53f74736 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1f8eef78b13e4f2783fdd769a602e202]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 08:56:09.885049 1922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aa5581fb8b34901afeac1ca53f74736. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:09.885155 2010 tablet_bootstrap.cc:654] T 1aa5581fb8b34901afeac1ca53f74736 P d1d8ba2d1e124bec8b100bfb4d2df3f6: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:09.885202 2008 tablet_bootstrap.cc:654] T 1aa5581fb8b34901afeac1ca53f74736 P 0d8681ac3160473f9c09472f2b3ec4c5: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:09.885720 2010 tablet_bootstrap.cc:492] T 1aa5581fb8b34901afeac1ca53f74736 P d1d8ba2d1e124bec8b100bfb4d2df3f6: No bootstrap required, opened a new log
I20260619 08:56:09.885774 2010 ts_tablet_manager.cc:1397] T 1aa5581fb8b34901afeac1ca53f74736 P d1d8ba2d1e124bec8b100bfb4d2df3f6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:09.885890 2010 raft_consensus.cc:348] T 1aa5581fb8b34901afeac1ca53f74736 P d1d8ba2d1e124bec8b100bfb4d2df3f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } }
I20260619 08:56:09.885962 2010 raft_consensus.cc:374] T 1aa5581fb8b34901afeac1ca53f74736 P d1d8ba2d1e124bec8b100bfb4d2df3f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:09.885979 2010 raft_consensus.cc:729] T 1aa5581fb8b34901afeac1ca53f74736 P d1d8ba2d1e124bec8b100bfb4d2df3f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1d8ba2d1e124bec8b100bfb4d2df3f6, State: Initialized, Role: FOLLOWER
I20260619 08:56:09.886024 2010 consensus_queue.cc:260] T 1aa5581fb8b34901afeac1ca53f74736 P d1d8ba2d1e124bec8b100bfb4d2df3f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } }
I20260619 08:56:09.886111 2010 ts_tablet_manager.cc:1428] T 1aa5581fb8b34901afeac1ca53f74736 P d1d8ba2d1e124bec8b100bfb4d2df3f6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:09.886142 2018 tablet_replica.cc:439] T 1aa5581fb8b34901afeac1ca53f74736 P d1d8ba2d1e124bec8b100bfb4d2df3f6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } } }
I20260619 08:56:09.886240 2018 tablet_replica.cc:442] T 1aa5581fb8b34901afeac1ca53f74736 P d1d8ba2d1e124bec8b100bfb4d2df3f6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:56:09.886474 2009 tablet_bootstrap.cc:492] T 1aa5581fb8b34901afeac1ca53f74736 P 3d20be2a71344f39bc23a1eef0517831: Bootstrap starting.
I20260619 08:56:09.886488 2008 tablet_bootstrap.cc:492] T 1aa5581fb8b34901afeac1ca53f74736 P 0d8681ac3160473f9c09472f2b3ec4c5: No bootstrap required, opened a new log
I20260619 08:56:09.886534 2008 ts_tablet_manager.cc:1397] T 1aa5581fb8b34901afeac1ca53f74736 P 0d8681ac3160473f9c09472f2b3ec4c5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:56:09.886675 2008 raft_consensus.cc:348] T 1aa5581fb8b34901afeac1ca53f74736 P 0d8681ac3160473f9c09472f2b3ec4c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } }
I20260619 08:56:09.886734 2008 raft_consensus.cc:374] T 1aa5581fb8b34901afeac1ca53f74736 P 0d8681ac3160473f9c09472f2b3ec4c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:09.886755 2008 raft_consensus.cc:729] T 1aa5581fb8b34901afeac1ca53f74736 P 0d8681ac3160473f9c09472f2b3ec4c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d8681ac3160473f9c09472f2b3ec4c5, State: Initialized, Role: FOLLOWER
I20260619 08:56:09.886809 2008 consensus_queue.cc:260] T 1aa5581fb8b34901afeac1ca53f74736 P 0d8681ac3160473f9c09472f2b3ec4c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } }
I20260619 08:56:09.886881 2009 tablet_bootstrap.cc:654] T 1aa5581fb8b34901afeac1ca53f74736 P 3d20be2a71344f39bc23a1eef0517831: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:09.886911 2008 ts_tablet_manager.cc:1428] T 1aa5581fb8b34901afeac1ca53f74736 P 0d8681ac3160473f9c09472f2b3ec4c5: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:09.887046 2015 tablet_replica.cc:439] T 1aa5581fb8b34901afeac1ca53f74736 P 0d8681ac3160473f9c09472f2b3ec4c5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } } }
I20260619 08:56:09.887118 2015 tablet_replica.cc:442] T 1aa5581fb8b34901afeac1ca53f74736 P 0d8681ac3160473f9c09472f2b3ec4c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:56:09.887526 2009 tablet_bootstrap.cc:492] T 1aa5581fb8b34901afeac1ca53f74736 P 3d20be2a71344f39bc23a1eef0517831: No bootstrap required, opened a new log
I20260619 08:56:09.887578 2009 ts_tablet_manager.cc:1397] T 1aa5581fb8b34901afeac1ca53f74736 P 3d20be2a71344f39bc23a1eef0517831: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:09.887712 2009 raft_consensus.cc:348] T 1aa5581fb8b34901afeac1ca53f74736 P 3d20be2a71344f39bc23a1eef0517831 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } }
I20260619 08:56:09.887782 2009 raft_consensus.cc:374] T 1aa5581fb8b34901afeac1ca53f74736 P 3d20be2a71344f39bc23a1eef0517831 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:09.887800 2009 raft_consensus.cc:729] T 1aa5581fb8b34901afeac1ca53f74736 P 3d20be2a71344f39bc23a1eef0517831 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d20be2a71344f39bc23a1eef0517831, State: Initialized, Role: FOLLOWER
I20260619 08:56:09.887841 2009 consensus_queue.cc:260] T 1aa5581fb8b34901afeac1ca53f74736 P 3d20be2a71344f39bc23a1eef0517831 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } }
I20260619 08:56:09.887924 2009 ts_tablet_manager.cc:1428] T 1aa5581fb8b34901afeac1ca53f74736 P 3d20be2a71344f39bc23a1eef0517831: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:09.887996 2016 tablet_replica.cc:439] T 1aa5581fb8b34901afeac1ca53f74736 P 3d20be2a71344f39bc23a1eef0517831: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } } }
I20260619 08:56:09.888144 2016 tablet_replica.cc:442] T 1aa5581fb8b34901afeac1ca53f74736 P 3d20be2a71344f39bc23a1eef0517831: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:56:10.015391 2016 raft_consensus.cc:482] T 1aa5581fb8b34901afeac1ca53f74736 P 3d20be2a71344f39bc23a1eef0517831 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:56:10.015512 2016 raft_consensus.cc:504] T 1aa5581fb8b34901afeac1ca53f74736 P 3d20be2a71344f39bc23a1eef0517831 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } }
I20260619 08:56:10.015857 2016 leader_election.cc:288] T 1aa5581fb8b34901afeac1ca53f74736 P 3d20be2a71344f39bc23a1eef0517831 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1d8ba2d1e124bec8b100bfb4d2df3f6 (127.0.98.1:45869), 0d8681ac3160473f9c09472f2b3ec4c5 (127.0.98.2:44941)
I20260619 08:56:10.019785 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:10.016150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39989 (local address 127.0.98.2:44941)
0619 08:56:10.016211 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:10.016213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:10.016444 (+ 231us) server_negotiation.cc:407] Connection header received
0619 08:56:10.016471 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:10.016472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:10.016481 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:10.016531 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:10.016660 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:10.017404 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:10.018860 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:10.019409 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:10.019477 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:10.019489 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:10.019664 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 08:56:10.019705 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 08:56:10.019791 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:10.016148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42727 (local address 127.0.98.1:45869)
0619 08:56:10.016209 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:10.016211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:10.016342 (+ 131us) server_negotiation.cc:407] Connection header received
0619 08:56:10.016399 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:10.016400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:10.016412 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:10.016510 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:10.016661 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:10.017399 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:10.018865 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:10.019409 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:10.019477 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:10.019495 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:10.019664 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 08:56:10.019716 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 08:56:10.019827 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:10.016057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44941 (local address 127.0.98.3:39989)
0619 08:56:10.016376 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:10.016385 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:10.016434 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:10.016537 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:10.016538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:10.016539 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:10.016597 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:10.016599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:10.017493 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:10.017495 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:10.018770 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:10.018774 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:10.019488 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:10.019489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:10.019585 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:10.019591 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 08:56:10.019651 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 08:56:10.019710 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":104,"thread_start_us":96,"threads_started":1}
I20260619 08:56:10.019934 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:10.016057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45869 (local address 127.0.98.3:42727)
0619 08:56:10.016266 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:10.016280 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:10.016339 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:10.016514 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:10.016515 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:10.016517 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:10.016602 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:10.016603 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:10.017529 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:10.017529 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:10.018770 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:10.018774 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:10.019525 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:10.019526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:10.019593 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:10.019595 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:56:10.019651 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 08:56:10.019716 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":9,"thread_start_us":141,"threads_started":1}
I20260619 08:56:10.020316 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aa5581fb8b34901afeac1ca53f74736" candidate_uuid: "3d20be2a71344f39bc23a1eef0517831" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" is_pre_election: true
I20260619 08:56:10.020411 1662 raft_consensus.cc:2454] T 1aa5581fb8b34901afeac1ca53f74736 P d1d8ba2d1e124bec8b100bfb4d2df3f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d20be2a71344f39bc23a1eef0517831 in term 0.
I20260619 08:56:10.020303 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aa5581fb8b34901afeac1ca53f74736" candidate_uuid: "3d20be2a71344f39bc23a1eef0517831" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" is_pre_election: true
I20260619 08:56:10.020498 1801 raft_consensus.cc:2454] T 1aa5581fb8b34901afeac1ca53f74736 P 0d8681ac3160473f9c09472f2b3ec4c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d20be2a71344f39bc23a1eef0517831 in term 0.
I20260619 08:56:10.020607 1875 leader_election.cc:302] T 1aa5581fb8b34901afeac1ca53f74736 P 3d20be2a71344f39bc23a1eef0517831 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d20be2a71344f39bc23a1eef0517831, d1d8ba2d1e124bec8b100bfb4d2df3f6; no voters:
I20260619 08:56:10.020756 2016 raft_consensus.cc:2781] T 1aa5581fb8b34901afeac1ca53f74736 P 3d20be2a71344f39bc23a1eef0517831 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:56:10.020798 2016 raft_consensus.cc:482] T 1aa5581fb8b34901afeac1ca53f74736 P 3d20be2a71344f39bc23a1eef0517831 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:56:10.020824 2016 raft_consensus.cc:3037] T 1aa5581fb8b34901afeac1ca53f74736 P 3d20be2a71344f39bc23a1eef0517831 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:10.021489 2016 raft_consensus.cc:504] T 1aa5581fb8b34901afeac1ca53f74736 P 3d20be2a71344f39bc23a1eef0517831 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } }
I20260619 08:56:10.021618 2016 leader_election.cc:288] T 1aa5581fb8b34901afeac1ca53f74736 P 3d20be2a71344f39bc23a1eef0517831 [CANDIDATE]: Term 1 election: Requested vote from peers d1d8ba2d1e124bec8b100bfb4d2df3f6 (127.0.98.1:45869), 0d8681ac3160473f9c09472f2b3ec4c5 (127.0.98.2:44941)
I20260619 08:56:10.021799 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aa5581fb8b34901afeac1ca53f74736" candidate_uuid: "3d20be2a71344f39bc23a1eef0517831" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d8681ac3160473f9c09472f2b3ec4c5"
I20260619 08:56:10.021811 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aa5581fb8b34901afeac1ca53f74736" candidate_uuid: "3d20be2a71344f39bc23a1eef0517831" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6"
I20260619 08:56:10.021888 1662 raft_consensus.cc:3037] T 1aa5581fb8b34901afeac1ca53f74736 P d1d8ba2d1e124bec8b100bfb4d2df3f6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:10.021888 1801 raft_consensus.cc:3037] T 1aa5581fb8b34901afeac1ca53f74736 P 0d8681ac3160473f9c09472f2b3ec4c5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:10.022656 1801 raft_consensus.cc:2454] T 1aa5581fb8b34901afeac1ca53f74736 P 0d8681ac3160473f9c09472f2b3ec4c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d20be2a71344f39bc23a1eef0517831 in term 1.
I20260619 08:56:10.022665 1662 raft_consensus.cc:2454] T 1aa5581fb8b34901afeac1ca53f74736 P d1d8ba2d1e124bec8b100bfb4d2df3f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d20be2a71344f39bc23a1eef0517831 in term 1.
I20260619 08:56:10.022836 1873 leader_election.cc:302] T 1aa5581fb8b34901afeac1ca53f74736 P 3d20be2a71344f39bc23a1eef0517831 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d8681ac3160473f9c09472f2b3ec4c5, 3d20be2a71344f39bc23a1eef0517831; no voters:
I20260619 08:56:10.022926 2016 raft_consensus.cc:2781] T 1aa5581fb8b34901afeac1ca53f74736 P 3d20be2a71344f39bc23a1eef0517831 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:10.023039 2016 raft_consensus.cc:686] T 1aa5581fb8b34901afeac1ca53f74736 P 3d20be2a71344f39bc23a1eef0517831 [term 1 LEADER]: Becoming Leader. State: Replica: 3d20be2a71344f39bc23a1eef0517831, State: Running, Role: LEADER
I20260619 08:56:10.023105 2016 consensus_queue.cc:237] T 1aa5581fb8b34901afeac1ca53f74736 P 3d20be2a71344f39bc23a1eef0517831 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } }
I20260619 08:56:10.023420 2051 tablet_replica.cc:439] T 1aa5581fb8b34901afeac1ca53f74736 P 3d20be2a71344f39bc23a1eef0517831: TxnStatusTablet state changed. Reason: New leader 3d20be2a71344f39bc23a1eef0517831. Latest consensus state: current_term: 1 leader_uuid: "3d20be2a71344f39bc23a1eef0517831" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } } }
I20260619 08:56:10.023510 2051 tablet_replica.cc:442] T 1aa5581fb8b34901afeac1ca53f74736 P 3d20be2a71344f39bc23a1eef0517831: This TxnStatusTablet replica's current role is: LEADER
I20260619 08:56:10.023675 2053 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 08:56:10.023794 2053 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 08:56:10.023766 1504 catalog_manager.cc:5495] T 1aa5581fb8b34901afeac1ca53f74736 P 3d20be2a71344f39bc23a1eef0517831 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d20be2a71344f39bc23a1eef0517831 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3d20be2a71344f39bc23a1eef0517831" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } health_report { overall_health: HEALTHY } } }
I20260619 08:56:10.032195 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:10.029539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33541 (local address 127.0.0.1:40948)
0619 08:56:10.029604 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:10.029615 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:10.029659 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:10.029825 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:10.029825 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:10.029826 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 08:56:10.029871 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:10.029873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:10.030741 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:10.030742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:10.031302 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:10.031304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:10.031733 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:10.031734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:10.031776 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:10.031789 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 08:56:10.032030 (+ 241us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 08:56:10.032034 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:56:10.032054 (+ 20us) client_negotiation.cc:240] Negotiation successful
0619 08:56:10.032121 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 08:56:10.032208 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:10.029607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40948 (local address 127.0.98.3:33541)
0619 08:56:10.029650 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:10.029652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:10.029662 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:56:10.029725 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:10.029726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:10.029745 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:10.029826 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:56:10.029922 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:10.030661 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:10.031380 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:10.031669 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:10.031739 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:10.031860 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:56:10.031975 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:56:10.032043 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:10.032066 (+ 23us) server_negotiation.cc:299] Negotiation successful
0619 08:56:10.032116 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 08:56:10.033424 1801 raft_consensus.cc:1261] T 1aa5581fb8b34901afeac1ca53f74736 P 0d8681ac3160473f9c09472f2b3ec4c5 [term 1 FOLLOWER]: Refusing update from remote peer 3d20be2a71344f39bc23a1eef0517831: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:56:10.033457 1662 raft_consensus.cc:1261] T 1aa5581fb8b34901afeac1ca53f74736 P d1d8ba2d1e124bec8b100bfb4d2df3f6 [term 1 FOLLOWER]: Refusing update from remote peer 3d20be2a71344f39bc23a1eef0517831: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:56:10.033689 2016 consensus_queue.cc:1035] T 1aa5581fb8b34901afeac1ca53f74736 P 3d20be2a71344f39bc23a1eef0517831 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:56:10.033793 2051 consensus_queue.cc:1035] T 1aa5581fb8b34901afeac1ca53f74736 P 3d20be2a71344f39bc23a1eef0517831 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:56:10.034412 2018 tablet_replica.cc:439] T 1aa5581fb8b34901afeac1ca53f74736 P d1d8ba2d1e124bec8b100bfb4d2df3f6: TxnStatusTablet state changed. Reason: New leader 3d20be2a71344f39bc23a1eef0517831. Latest consensus state: current_term: 1 leader_uuid: "3d20be2a71344f39bc23a1eef0517831" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } } }
I20260619 08:56:10.034509 2018 tablet_replica.cc:442] T 1aa5581fb8b34901afeac1ca53f74736 P d1d8ba2d1e124bec8b100bfb4d2df3f6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:56:10.034655 2015 tablet_replica.cc:439] T 1aa5581fb8b34901afeac1ca53f74736 P 0d8681ac3160473f9c09472f2b3ec4c5: TxnStatusTablet state changed. Reason: New leader 3d20be2a71344f39bc23a1eef0517831. Latest consensus state: current_term: 1 leader_uuid: "3d20be2a71344f39bc23a1eef0517831" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } } }
I20260619 08:56:10.034739 2015 tablet_replica.cc:442] T 1aa5581fb8b34901afeac1ca53f74736 P 0d8681ac3160473f9c09472f2b3ec4c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:56:10.035324 2051 tablet_replica.cc:439] T 1aa5581fb8b34901afeac1ca53f74736 P 3d20be2a71344f39bc23a1eef0517831: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3d20be2a71344f39bc23a1eef0517831" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } } }
I20260619 08:56:10.035403 2051 tablet_replica.cc:442] T 1aa5581fb8b34901afeac1ca53f74736 P 3d20be2a71344f39bc23a1eef0517831: This TxnStatusTablet replica's current role is: LEADER
I20260619 08:56:10.035564 2016 tablet_replica.cc:439] T 1aa5581fb8b34901afeac1ca53f74736 P 3d20be2a71344f39bc23a1eef0517831: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3d20be2a71344f39bc23a1eef0517831" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } } }
I20260619 08:56:10.035637 2016 tablet_replica.cc:442] T 1aa5581fb8b34901afeac1ca53f74736 P 3d20be2a71344f39bc23a1eef0517831: This TxnStatusTablet replica's current role is: LEADER
I20260619 08:56:10.035920 2018 tablet_replica.cc:439] T 1aa5581fb8b34901afeac1ca53f74736 P d1d8ba2d1e124bec8b100bfb4d2df3f6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3d20be2a71344f39bc23a1eef0517831" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } } }
I20260619 08:56:10.036000 2018 tablet_replica.cc:442] T 1aa5581fb8b34901afeac1ca53f74736 P d1d8ba2d1e124bec8b100bfb4d2df3f6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:56:10.036305 2015 tablet_replica.cc:439] T 1aa5581fb8b34901afeac1ca53f74736 P 0d8681ac3160473f9c09472f2b3ec4c5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3d20be2a71344f39bc23a1eef0517831" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 } } peers { permanent_uuid: "0d8681ac3160473f9c09472f2b3ec4c5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44941 } } peers { permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 } } }
I20260619 08:56:10.036391 2015 tablet_replica.cc:442] T 1aa5581fb8b34901afeac1ca53f74736 P 0d8681ac3160473f9c09472f2b3ec4c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:56:10.039278 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:10.037085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39942 (local address 127.0.98.2:44941)
0619 08:56:10.037123 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:10.037124 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:10.037132 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:56:10.037195 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:10.037196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:10.037214 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:10.037273 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:56:10.037368 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:10.037827 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:10.038585 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:10.038784 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:10.038836 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:10.038972 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:56:10.039076 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:56:10.039137 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:10.039182 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 08:56:10.039200 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 08:56:10.044595 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:10.041655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44599 (local address 127.0.98.3:33541)
0619 08:56:10.041696 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:10.041698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:10.041767 (+ 69us) server_negotiation.cc:407] Connection header received
0619 08:56:10.041820 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:10.041821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:10.041828 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:10.041869 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:56:10.042001 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:10.042428 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:10.043971 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:10.044197 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:10.044244 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:10.044420 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:56:10.044444 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:56:10.044476 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:10.044528 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 08:56:10.044545 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 08:56:10.044658 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:10.041581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33541 (local address 127.0.98.2:44599)
0619 08:56:10.041694 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:10.041705 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:10.041765 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:10.041874 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:10.041875 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:10.041876 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 08:56:10.041950 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:10.041951 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:10.042486 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:10.042486 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:10.043889 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:10.043891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:10.044253 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:10.044254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:10.044319 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:10.044333 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 08:56:10.044478 (+ 145us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 08:56:10.044479 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 08:56:10.044526 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 08:56:10.044568 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":55,"threads_started":1}
I20260619 08:56:10.046828 1662 raft_consensus.cc:1261] T 7b96429d9e434ec1bff8ca110e3ef1fa P d1d8ba2d1e124bec8b100bfb4d2df3f6 [term 1 FOLLOWER]: Refusing update from remote peer 0d8681ac3160473f9c09472f2b3ec4c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:56:10.046888 1942 raft_consensus.cc:1261] T 7b96429d9e434ec1bff8ca110e3ef1fa P 3d20be2a71344f39bc23a1eef0517831 [term 1 FOLLOWER]: Refusing update from remote peer 0d8681ac3160473f9c09472f2b3ec4c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:56:10.047096 2015 consensus_queue.cc:1035] T 7b96429d9e434ec1bff8ca110e3ef1fa P 0d8681ac3160473f9c09472f2b3ec4c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d20be2a71344f39bc23a1eef0517831" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:56:10.047271 2038 consensus_queue.cc:1035] T 7b96429d9e434ec1bff8ca110e3ef1fa P 0d8681ac3160473f9c09472f2b3ec4c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1d8ba2d1e124bec8b100bfb4d2df3f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:56:10.057070 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:10.053846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44941 (local address 127.0.98.3:42993)
0619 08:56:10.054014 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:10.054025 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:10.054082 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:10.054201 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:10.054202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:10.054214 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 08:56:10.054274 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:10.054276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:10.055143 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:10.055144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:10.056018 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:10.056021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:10.056578 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:10.056583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:10.056641 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:10.056660 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 08:56:10.056880 (+ 220us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 08:56:10.056881 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 08:56:10.056912 (+ 31us) client_negotiation.cc:240] Negotiation successful
0619 08:56:10.056952 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":80,"threads_started":1}
I20260619 08:56:10.057063 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:10.053983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42993 (local address 127.0.98.2:44941)
0619 08:56:10.054020 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:10.054022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:10.054073 (+ 51us) server_negotiation.cc:407] Connection header received
0619 08:56:10.054131 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:10.054132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:10.054143 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:10.054199 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:56:10.054326 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:10.055047 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:10.056119 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:10.056476 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:10.056550 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:10.056769 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:56:10.056813 (+ 44us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:56:10.056880 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:10.056934 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 08:56:10.056959 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 08:56:10.067318 1505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:56:10.067446 1505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:56:10.068120 1505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4aa76475773049539f1aaa539293fd94: Sending DeleteTablet for 3 replicas of tablet 7b96429d9e434ec1bff8ca110e3ef1fa
I20260619 08:56:10.068459 1642 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b96429d9e434ec1bff8ca110e3ef1fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33712
I20260619 08:56:10.068589 2033 tablet_replica.cc:330] T 7b96429d9e434ec1bff8ca110e3ef1fa P d1d8ba2d1e124bec8b100bfb4d2df3f6: stopping tablet replica
I20260619 08:56:10.068595 1922 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b96429d9e434ec1bff8ca110e3ef1fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40940
I20260619 08:56:10.068660 2033 raft_consensus.cc:2229] T 7b96429d9e434ec1bff8ca110e3ef1fa P d1d8ba2d1e124bec8b100bfb4d2df3f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:56:10.068704 2033 raft_consensus.cc:2258] T 7b96429d9e434ec1bff8ca110e3ef1fa P d1d8ba2d1e124bec8b100bfb4d2df3f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:10.068778 2031 tablet_replica.cc:330] T 7b96429d9e434ec1bff8ca110e3ef1fa P 3d20be2a71344f39bc23a1eef0517831: stopping tablet replica
I20260619 08:56:10.068833 2031 raft_consensus.cc:2229] T 7b96429d9e434ec1bff8ca110e3ef1fa P 3d20be2a71344f39bc23a1eef0517831 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:56:10.068866 2031 raft_consensus.cc:2258] T 7b96429d9e434ec1bff8ca110e3ef1fa P 3d20be2a71344f39bc23a1eef0517831 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:10.068899 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b96429d9e434ec1bff8ca110e3ef1fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39930
I20260619 08:56:10.068998 2032 tablet_replica.cc:330] T 7b96429d9e434ec1bff8ca110e3ef1fa P 0d8681ac3160473f9c09472f2b3ec4c5: stopping tablet replica
I20260619 08:56:10.069061 2032 raft_consensus.cc:2229] T 7b96429d9e434ec1bff8ca110e3ef1fa P 0d8681ac3160473f9c09472f2b3ec4c5 [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:56:10.069079 2033 ts_tablet_manager.cc:1905] T 7b96429d9e434ec1bff8ca110e3ef1fa P d1d8ba2d1e124bec8b100bfb4d2df3f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:10.069151 2032 raft_consensus.cc:2258] T 7b96429d9e434ec1bff8ca110e3ef1fa P 0d8681ac3160473f9c09472f2b3ec4c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:10.069442 2031 ts_tablet_manager.cc:1905] T 7b96429d9e434ec1bff8ca110e3ef1fa P 3d20be2a71344f39bc23a1eef0517831: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:10.069478 2032 ts_tablet_manager.cc:1905] T 7b96429d9e434ec1bff8ca110e3ef1fa P 0d8681ac3160473f9c09472f2b3ec4c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:10.070787 2033 ts_tablet_manager.cc:1918] T 7b96429d9e434ec1bff8ca110e3ef1fa P d1d8ba2d1e124bec8b100bfb4d2df3f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 08:56:10.070847 2033 log.cc:1196] T 7b96429d9e434ec1bff8ca110e3ef1fa P d1d8ba2d1e124bec8b100bfb4d2df3f6: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-0/wal/wals/7b96429d9e434ec1bff8ca110e3ef1fa
I20260619 08:56:10.071009 2031 ts_tablet_manager.cc:1918] T 7b96429d9e434ec1bff8ca110e3ef1fa P 3d20be2a71344f39bc23a1eef0517831: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 08:56:10.071055 2031 log.cc:1196] T 7b96429d9e434ec1bff8ca110e3ef1fa P 3d20be2a71344f39bc23a1eef0517831: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-2/wal/wals/7b96429d9e434ec1bff8ca110e3ef1fa
I20260619 08:56:10.071077 2033 ts_tablet_manager.cc:1939] T 7b96429d9e434ec1bff8ca110e3ef1fa P d1d8ba2d1e124bec8b100bfb4d2df3f6: Deleting consensus metadata
I20260619 08:56:10.071219 2031 ts_tablet_manager.cc:1939] T 7b96429d9e434ec1bff8ca110e3ef1fa P 3d20be2a71344f39bc23a1eef0517831: Deleting consensus metadata
I20260619 08:56:10.071296 2032 ts_tablet_manager.cc:1918] T 7b96429d9e434ec1bff8ca110e3ef1fa P 0d8681ac3160473f9c09472f2b3ec4c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 08:56:10.071368 2032 log.cc:1196] T 7b96429d9e434ec1bff8ca110e3ef1fa P 0d8681ac3160473f9c09472f2b3ec4c5: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859361540516-392-0/minicluster-data/ts-1/wal/wals/7b96429d9e434ec1bff8ca110e3ef1fa
I20260619 08:56:10.071556 2032 ts_tablet_manager.cc:1939] T 7b96429d9e434ec1bff8ca110e3ef1fa P 0d8681ac3160473f9c09472f2b3ec4c5: Deleting consensus metadata
I20260619 08:56:10.071550 1491 catalog_manager.cc:4841] TS 3d20be2a71344f39bc23a1eef0517831 (127.0.98.3:33541): tablet 7b96429d9e434ec1bff8ca110e3ef1fa (table test-table [id=645c3ee4a21a4d2b8f87d8bb04676f12]) successfully deleted
I20260619 08:56:10.071893 1489 catalog_manager.cc:4841] TS 0d8681ac3160473f9c09472f2b3ec4c5 (127.0.98.2:44941): tablet 7b96429d9e434ec1bff8ca110e3ef1fa (table test-table [id=645c3ee4a21a4d2b8f87d8bb04676f12]) successfully deleted
I20260619 08:56:10.072145 1491 catalog_manager.cc:4841] TS d1d8ba2d1e124bec8b100bfb4d2df3f6 (127.0.98.1:45869): tablet 7b96429d9e434ec1bff8ca110e3ef1fa (table test-table [id=645c3ee4a21a4d2b8f87d8bb04676f12]) successfully deleted
Jun 19 08:56:10 dist-test-slave-h4g5 krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859369, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 08:56:10.083693 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:10.071798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33734 (local address 127.0.98.1:45869)
0619 08:56:10.071842 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:10.071844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:10.072647 (+ 803us) server_negotiation.cc:407] Connection header received
0619 08:56:10.072751 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:10.072752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:10.072762 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:10.072790 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:10.073606 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:10.074360 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:10.075681 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:10.076017 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:10.076066 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:10.080115 (+ 4049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:10.080130 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:10.080130 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:10.080139 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:10.082241 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:10.082707 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:10.082712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:10.082712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:10.082776 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:10.082965 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:10.082966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:10.082966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:10.083159 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:10.083284 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:10.083495 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 08:56:10.083586 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260619 08:56:10.084401 1696 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33734
I20260619 08:56:10.093580 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:10.085411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36984 (local address 127.0.98.62:38807)
0619 08:56:10.085456 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:10.085457 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:10.085710 (+ 253us) server_negotiation.cc:407] Connection header received
0619 08:56:10.085783 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:10.085784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:10.085797 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:10.085859 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:10.086542 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:10.087348 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:10.088449 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:10.088762 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:10.088837 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:10.090538 (+ 1701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:10.090539 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:10.090540 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:10.090548 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:10.092333 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:10.092755 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:10.092757 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:10.092757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:10.092821 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:10.093027 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:10.093028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:10.093028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:10.093166 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:10.093260 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:10.093394 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 08:56:10.093475 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260619 08:56:10.094190 1505 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36984
I20260619 08:56:10.094935 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 1577
I20260619 08:56:10.104010 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 1717
I20260619 08:56:10.108956 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 1857
I20260619 08:56:10.113485 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 1473
2026-06-19T08:56:10Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2329 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:56:10 dist-test-slave-h4g5 krb5kdc[2075](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:56:10 dist-test-slave-h4g5 krb5kdc[2075](info): set up 2 sockets
Jun 19 08:56:10 dist-test-slave-h4g5 krb5kdc[2075](info): commencing operation
krb5kdc: starting...
W20260619 08:56:11.063620 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 08:56:11 dist-test-slave-h4g5 krb5kdc[2075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:56:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:56:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 08:56:11.179013 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43031
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:11.257718 2088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:11.257929 2088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:11.257958 2088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:11.259105 2088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:56:11.259146 2088 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 08:56:11.259163 2088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:11.259181 2088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:11.259196 2088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:56:11.259213 2088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:56:11.260704 2088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43031
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:11.260946 2088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:11.261157 2088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:11.265895 2094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:11.265897 2096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:11.265890 2093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:11.266191 2088 server_base.cc:1034] running on GCE node
I20260619 08:56:11.266402 2088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:11.266673 2088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:11.267844 2088 hybrid_clock.cc:648] HybridClock initialized: now 1781859371267824 us; error 27 us; skew 500 ppm
Jun 19 08:56:11 dist-test-slave-h4g5 krb5kdc[2075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:11.273885 2088 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 08:56:11.274340 2088 webserver.cc:460] Webserver started at http://127.0.98.62:34323/ using document root <none> and password file <none>
I20260619 08:56:11.274564 2088 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:11.274623 2088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:11.274749 2088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:11.275800 2088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "5efeda2ae7b6428aada15b2780867507"
format_stamp: "Formatted at 2026-06-19 08:56:11 on dist-test-slave-h4g5"
server_key: "5bd1776cab89bab322d92fe60b6d1439"
server_key_iv: "11a2e21f1b6914faff8083b719e6f81f"
server_key_version: "encryptionkey@0"
I20260619 08:56:11.276183 2088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "5efeda2ae7b6428aada15b2780867507"
format_stamp: "Formatted at 2026-06-19 08:56:11 on dist-test-slave-h4g5"
server_key: "5bd1776cab89bab322d92fe60b6d1439"
server_key_iv: "11a2e21f1b6914faff8083b719e6f81f"
server_key_version: "encryptionkey@0"
I20260619 08:56:11.277514 2088 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:11.278249 2103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:11.278414 2088 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:11.278497 2088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "5efeda2ae7b6428aada15b2780867507"
format_stamp: "Formatted at 2026-06-19 08:56:11 on dist-test-slave-h4g5"
server_key: "5bd1776cab89bab322d92fe60b6d1439"
server_key_iv: "11a2e21f1b6914faff8083b719e6f81f"
server_key_version: "encryptionkey@0"
I20260619 08:56:11.278568 2088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:11.300005 2088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:11.302155 2088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:11.302268 2088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:11.306524 2088 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43031
I20260619 08:56:11.306556 2165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43031 every 8 connection(s)
I20260619 08:56:11.306898 2088 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:56:11.307612 2166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:11.309954 2166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5efeda2ae7b6428aada15b2780867507: Bootstrap starting.
I20260619 08:56:11.310528 2166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5efeda2ae7b6428aada15b2780867507: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:11.310786 2166 log.cc:824] T 00000000000000000000000000000000 P 5efeda2ae7b6428aada15b2780867507: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:11.311589 2166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5efeda2ae7b6428aada15b2780867507: No bootstrap required, opened a new log
I20260619 08:56:11.313752 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 2088
I20260619 08:56:11.313841 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 08:56:11.314003 392 external_mini_cluster.cc:1428] Setting key 71fb5d4681a3909908f305cc21473e13
I20260619 08:56:11.314777 2166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5efeda2ae7b6428aada15b2780867507 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5efeda2ae7b6428aada15b2780867507" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43031 } }
I20260619 08:56:11.315013 2166 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5efeda2ae7b6428aada15b2780867507 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:11.315088 2166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5efeda2ae7b6428aada15b2780867507 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5efeda2ae7b6428aada15b2780867507, State: Initialized, Role: FOLLOWER
I20260619 08:56:11.315227 2166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5efeda2ae7b6428aada15b2780867507 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5efeda2ae7b6428aada15b2780867507" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43031 } }
I20260619 08:56:11.315310 2166 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5efeda2ae7b6428aada15b2780867507 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:56:11.315340 2166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5efeda2ae7b6428aada15b2780867507 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:56:11.315407 2166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5efeda2ae7b6428aada15b2780867507 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:11.316183 2166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5efeda2ae7b6428aada15b2780867507 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5efeda2ae7b6428aada15b2780867507" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43031 } }
I20260619 08:56:11.316324 2166 leader_election.cc:302] T 00000000000000000000000000000000 P 5efeda2ae7b6428aada15b2780867507 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5efeda2ae7b6428aada15b2780867507; no voters:
I20260619 08:56:11.316499 2166 leader_election.cc:288] T 00000000000000000000000000000000 P 5efeda2ae7b6428aada15b2780867507 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:56:11.316530 2171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5efeda2ae7b6428aada15b2780867507 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:11.316740 2171 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5efeda2ae7b6428aada15b2780867507 [term 1 LEADER]: Becoming Leader. State: Replica: 5efeda2ae7b6428aada15b2780867507, State: Running, Role: LEADER
I20260619 08:56:11.316865 2166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5efeda2ae7b6428aada15b2780867507 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:56:11.316865 2171 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5efeda2ae7b6428aada15b2780867507 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5efeda2ae7b6428aada15b2780867507" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43031 } }
I20260619 08:56:11.317603 2171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5efeda2ae7b6428aada15b2780867507 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5efeda2ae7b6428aada15b2780867507" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5efeda2ae7b6428aada15b2780867507" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43031 } } }
I20260619 08:56:11.317759 2171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5efeda2ae7b6428aada15b2780867507 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:11.317924 2171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5efeda2ae7b6428aada15b2780867507 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5efeda2ae7b6428aada15b2780867507. Latest consensus state: current_term: 1 leader_uuid: "5efeda2ae7b6428aada15b2780867507" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5efeda2ae7b6428aada15b2780867507" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43031 } } }
I20260619 08:56:11.318034 2171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5efeda2ae7b6428aada15b2780867507 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:11.318388 2180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:56:11.319651 2180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 08:56:11 dist-test-slave-h4g5 krb5kdc[2075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:11.321321 2180 catalog_manager.cc:1347] Generated new cluster ID: 55a1d80e24c7457a894e60eb9f110869
I20260619 08:56:11.321391 2180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:56:11.324823 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:11.314602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33424 (local address 127.0.98.62:43031)
0619 08:56:11.314803 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:11.314813 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:11.314846 (+ 33us) server_negotiation.cc:407] Connection header received
0619 08:56:11.314979 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:11.314987 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:11.315059 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:11.315926 (+ 867us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:11.316055 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.317423 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.318631 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.319234 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.319326 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:11.321739 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:11.321743 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:11.321743 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:11.321757 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:11.323274 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:11.323587 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:11.323590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:11.323590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:11.323694 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:11.323869 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:11.323870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:11.323870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:11.324141 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:11.324287 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:11.324512 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 08:56:11.324653 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":94,"threads_started":1}
I20260619 08:56:11.330309 2180 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:56:11.331058 2180 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:56:11.337059 2180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5efeda2ae7b6428aada15b2780867507: Generated new TSK 0
I20260619 08:56:11.337247 2180 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 08:56:11.365101 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43031
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:11.446789 2192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:11.447010 2192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:11.447036 2192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:11.448194 2192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:11.448231 2192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:11.448251 2192 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 08:56:11.448261 2192 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 08:56:11.448276 2192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 08:56:11.449710 2192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:11.449905 2192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:11.450088 2192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:11.454964 2198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:11.454964 2200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:11.455049 2197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:11.455229 2192 server_base.cc:1034] running on GCE node
I20260619 08:56:11.455425 2192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:11.455679 2192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:11.456832 2192 hybrid_clock.cc:648] HybridClock initialized: now 1781859371456812 us; error 37 us; skew 500 ppm
Jun 19 08:56:11 dist-test-slave-h4g5 krb5kdc[2075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:11.459241 2192 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 08:56:11.459718 2192 webserver.cc:460] Webserver started at http://127.0.98.1:33263/ using document root <none> and password file <none>
I20260619 08:56:11.459926 2192 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:11.459969 2192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:11.460124 2192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:11.461170 2192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-0/data/instance:
uuid: "facf427fe9e541b781955151f2be5909"
format_stamp: "Formatted at 2026-06-19 08:56:11 on dist-test-slave-h4g5"
server_key: "9076e3271862659e8072ccc76627abd4"
server_key_iv: "901893b967ebddca3533832650a07187"
server_key_version: "encryptionkey@0"
I20260619 08:56:11.461503 2192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-0/wal/instance:
uuid: "facf427fe9e541b781955151f2be5909"
format_stamp: "Formatted at 2026-06-19 08:56:11 on dist-test-slave-h4g5"
server_key: "9076e3271862659e8072ccc76627abd4"
server_key_iv: "901893b967ebddca3533832650a07187"
server_key_version: "encryptionkey@0"
I20260619 08:56:11.462862 2192 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 08:56:11.463550 2207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:11.463688 2192 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 08:56:11.463768 2192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-0/wal
uuid: "facf427fe9e541b781955151f2be5909"
format_stamp: "Formatted at 2026-06-19 08:56:11 on dist-test-slave-h4g5"
server_key: "9076e3271862659e8072ccc76627abd4"
server_key_iv: "901893b967ebddca3533832650a07187"
server_key_version: "encryptionkey@0"
I20260619 08:56:11.463832 2192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:11.490253 2192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:11.492398 2192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:11.492549 2192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:11.493175 2192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:11.493233 2192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:11.493265 2192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:11.493280 2192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 08:56:11 dist-test-slave-h4g5 krb5kdc[2075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:11.499918 2192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34365
I20260619 08:56:11.499981 2322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34365 every 8 connection(s)
I20260619 08:56:11.500406 2192 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
I20260619 08:56:11.503091 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:11.493636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54625 (local address 127.0.98.62:43031)
0619 08:56:11.493702 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:11.493705 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:11.493937 (+ 232us) server_negotiation.cc:407] Connection header received
0619 08:56:11.494613 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:11.494614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:11.494630 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:11.494685 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:11.495557 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.496289 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.497813 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.498043 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.498111 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:11.500457 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:11.500458 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:11.500459 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:11.500470 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:11.501948 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:11.502289 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:11.502289 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:11.502290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:11.502332 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:11.502549 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:11.502549 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:11.502550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:11.502659 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:11.502711 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:11.502956 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 08:56:11.503027 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 08:56:11.503182 2216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:11.493600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43031 (local address 127.0.98.1:54625)
0619 08:56:11.493785 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:11.493836 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:11.494490 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:11.494771 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:11.494776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:11.495140 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:11.495453 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:11.495459 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.496364 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.496365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:11.497700 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:11.497703 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.498128 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.498129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:11.498267 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:11.498758 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:11.498768 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:11.500344 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:11.502013 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:11.502018 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:11.502020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:11.502234 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:11.502374 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:11.502375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:11.502375 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:11.502463 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:11.502708 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:11.502711 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:11.502822 (+ 111us) client_negotiation.cc:769] Sending connection context
0619 08:56:11.502901 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 08:56:11.503018 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":91,"threads_started":1}
I20260619 08:56:11.503746 2323 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43031
I20260619 08:56:11.503829 2323 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:11.504014 2323 heartbeater.cc:511] Master 127.0.98.62:43031 requested a full tablet report, sending...
I20260619 08:56:11.504606 2119 ts_manager.cc:194] Registered new tserver with Master: facf427fe9e541b781955151f2be5909 (127.0.98.1:34365)
I20260619 08:56:11.505285 2119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54625
I20260619 08:56:11.510111 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 2192
I20260619 08:56:11.510143 2329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:11.504735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43031 (local address 127.0.98.1:57731)
0619 08:56:11.504878 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:11.504889 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:11.504940 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:11.505057 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:11.505057 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:11.505219 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:11.505274 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:11.505276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.506161 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.506162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:11.506663 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:11.506665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.507044 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.507044 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:11.507103 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:11.507541 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:11.507546 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:11.507838 (+ 292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:11.509130 (+ 1292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:11.509131 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:11.509131 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:11.509303 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:11.509573 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:11.509574 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:11.509575 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:11.509638 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:11.509889 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:11.509889 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:11.509921 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 08:56:11.509977 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 08:56:11.510057 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
I20260619 08:56:11.510190 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:11.504906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57731 (local address 127.0.98.62:43031)
0619 08:56:11.504950 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:11.504952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:11.504959 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:56:11.504989 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:11.504990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:11.504999 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:11.505041 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:11.505340 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.506076 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.506779 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.506962 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.507027 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:11.507915 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:11.507916 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:11.507917 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:11.507922 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:11.509032 (+ 1110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:11.509394 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:11.509395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:11.509395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:11.509450 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:11.509721 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:11.509722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:11.509722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:11.509832 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:11.509897 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:11.510015 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 08:56:11.510089 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 08:56:11.510203 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-0/wal/instance
I20260619 08:56:11.510386 392 external_mini_cluster.cc:1428] Setting key ba5cc90d32484fb4aa58e6ed4c0d81fe
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 08:56:11.534947 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43031
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:11.615299 2332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:11.615512 2332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:11.615546 2332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:11.616734 2332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:11.616775 2332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:11.616802 2332 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 08:56:11.616816 2332 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 08:56:11.616839 2332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 08:56:11.618422 2332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:11.618656 2332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:11.618862 2332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:11.623579 2338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:11.623659 2337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:11.623602 2340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:11.623885 2332 server_base.cc:1034] running on GCE node
I20260619 08:56:11.624068 2332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:11.624341 2332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:11.625499 2332 hybrid_clock.cc:648] HybridClock initialized: now 1781859371625483 us; error 35 us; skew 500 ppm
Jun 19 08:56:11 dist-test-slave-h4g5 krb5kdc[2075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:11.628018 2332 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 08:56:11.628501 2332 webserver.cc:460] Webserver started at http://127.0.98.2:35433/ using document root <none> and password file <none>
I20260619 08:56:11.628705 2332 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:11.628753 2332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:11.628877 2332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:11.629906 2332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-1/data/instance:
uuid: "f2fad2d8cc29452d8cef0645665b4a8c"
format_stamp: "Formatted at 2026-06-19 08:56:11 on dist-test-slave-h4g5"
server_key: "a0939929b2d161a0b58036248bfa7a31"
server_key_iv: "327b9e9251c6399f48ac61a43c7d7ca7"
server_key_version: "encryptionkey@0"
I20260619 08:56:11.630249 2332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f2fad2d8cc29452d8cef0645665b4a8c"
format_stamp: "Formatted at 2026-06-19 08:56:11 on dist-test-slave-h4g5"
server_key: "a0939929b2d161a0b58036248bfa7a31"
server_key_iv: "327b9e9251c6399f48ac61a43c7d7ca7"
server_key_version: "encryptionkey@0"
I20260619 08:56:11.631484 2332 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:56:11.632253 2347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:11.632426 2332 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:56:11.632511 2332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-1/wal
uuid: "f2fad2d8cc29452d8cef0645665b4a8c"
format_stamp: "Formatted at 2026-06-19 08:56:11 on dist-test-slave-h4g5"
server_key: "a0939929b2d161a0b58036248bfa7a31"
server_key_iv: "327b9e9251c6399f48ac61a43c7d7ca7"
server_key_version: "encryptionkey@0"
I20260619 08:56:11.632587 2332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:11.650105 2332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:11.652432 2332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:11.652558 2332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:11.653091 2332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:11.653137 2332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:11.653169 2332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:11.653185 2332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:11.659559 2332 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34599
I20260619 08:56:11.659698 2462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34599 every 8 connection(s)
Jun 19 08:56:11 dist-test-slave-h4g5 krb5kdc[2075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:11.659963 2332 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
I20260619 08:56:11.663025 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:11.653556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49237 (local address 127.0.98.62:43031)
0619 08:56:11.653604 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:11.653605 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:11.653836 (+ 231us) server_negotiation.cc:407] Connection header received
0619 08:56:11.654358 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:11.654360 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:11.654372 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:11.654423 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:11.655262 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.656008 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.657146 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.657341 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.657397 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:11.660487 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:11.660489 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:11.660489 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:11.660496 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:11.661754 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:11.662125 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:11.662126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:11.662126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:11.662186 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:11.662419 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:11.662420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:11.662421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:11.662553 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:11.662623 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:11.662896 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 08:56:11.662962 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 08:56:11.663177 2356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:11.653552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43031 (local address 127.0.98.2:49237)
0619 08:56:11.653720 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:11.653755 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:11.654259 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:11.654499 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:11.654504 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:11.654842 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:11.655143 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:11.655149 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.656137 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.656138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:11.657057 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:11.657059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.657428 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.657429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:11.657557 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:11.658543 (+ 986us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:11.658561 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:11.660401 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:11.661829 (+ 1428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:11.661835 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:11.661838 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:11.662048 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:11.662234 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:11.662234 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:11.662234 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:11.662327 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:11.662659 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:11.662665 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:11.662786 (+ 121us) client_negotiation.cc:769] Sending connection context
0619 08:56:11.662872 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 08:56:11.662997 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":100,"threads_started":1}
I20260619 08:56:11.663591 2463 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43031
I20260619 08:56:11.663707 2463 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:11.663895 2463 heartbeater.cc:511] Master 127.0.98.62:43031 requested a full tablet report, sending...
I20260619 08:56:11.664278 2119 ts_manager.cc:194] Registered new tserver with Master: f2fad2d8cc29452d8cef0645665b4a8c (127.0.98.2:34599)
I20260619 08:56:11.664719 2119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49237
I20260619 08:56:11.669675 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:11.664671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50219 (local address 127.0.98.62:43031)
0619 08:56:11.664717 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:11.664720 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:11.664748 (+ 28us) server_negotiation.cc:407] Connection header received
0619 08:56:11.664818 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:11.664819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:11.664828 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:11.664879 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:11.665205 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.665801 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.666476 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.666690 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.666741 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:11.667582 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:11.667584 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:11.667584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:11.667591 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:11.668691 (+ 1100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:11.668970 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:11.668971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:11.668971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:11.669021 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:11.669174 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:11.669175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:11.669175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:11.669310 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:11.669364 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:11.669497 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 08:56:11.669567 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 08:56:11.669683 2469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:11.664570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43031 (local address 127.0.98.2:50219)
0619 08:56:11.664696 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:11.664707 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:11.664762 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:11.664889 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:11.664890 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:11.665065 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:11.665139 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:11.665141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.665874 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.665874 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:11.666414 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:11.666416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.666749 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.666749 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:11.666826 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:11.667219 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:11.667224 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:11.667512 (+ 288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:11.668760 (+ 1248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:11.668761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:11.668762 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:11.668912 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:11.669070 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:11.669071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:11.669071 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:11.669125 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:11.669365 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:11.669366 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:11.669411 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 08:56:11.669470 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 08:56:11.669568 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":79,"threads_started":1}
I20260619 08:56:11.669829 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 2332
I20260619 08:56:11.669910 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-1/wal/instance
I20260619 08:56:11.670073 392 external_mini_cluster.cc:1428] Setting key 8ab9b30398fb4b8a9faa1c0ea1d0501b
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 08:56:11.695003 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43031
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:11.774261 2472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:11.774456 2472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:11.774487 2472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:11.775619 2472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:11.775660 2472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:11.775686 2472 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 08:56:11.775701 2472 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 08:56:11.775732 2472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 08:56:11.777192 2472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:11.777415 2472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:11.777611 2472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:11.782572 2477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:11.782546 2480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:11.782588 2478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:11.782812 2472 server_base.cc:1034] running on GCE node
I20260619 08:56:11.782981 2472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:11.783213 2472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:11.784360 2472 hybrid_clock.cc:648] HybridClock initialized: now 1781859371784347 us; error 29 us; skew 500 ppm
Jun 19 08:56:11 dist-test-slave-h4g5 krb5kdc[2075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:11.786756 2472 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 08:56:11.787180 2472 webserver.cc:460] Webserver started at http://127.0.98.3:38153/ using document root <none> and password file <none>
I20260619 08:56:11.787365 2472 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:11.787403 2472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:11.787492 2472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:11.788502 2472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-2/data/instance:
uuid: "1f909d146c124ed7849be361ca427112"
format_stamp: "Formatted at 2026-06-19 08:56:11 on dist-test-slave-h4g5"
server_key: "9e7fb46a6cca227cad087ecd4abf8ff6"
server_key_iv: "d1b55025eea70dc1f30750d2f446084c"
server_key_version: "encryptionkey@0"
I20260619 08:56:11.788786 2472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1f909d146c124ed7849be361ca427112"
format_stamp: "Formatted at 2026-06-19 08:56:11 on dist-test-slave-h4g5"
server_key: "9e7fb46a6cca227cad087ecd4abf8ff6"
server_key_iv: "d1b55025eea70dc1f30750d2f446084c"
server_key_version: "encryptionkey@0"
I20260619 08:56:11.789934 2472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:56:11.790601 2487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:11.790799 2472 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:11.790882 2472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-2/wal
uuid: "1f909d146c124ed7849be361ca427112"
format_stamp: "Formatted at 2026-06-19 08:56:11 on dist-test-slave-h4g5"
server_key: "9e7fb46a6cca227cad087ecd4abf8ff6"
server_key_iv: "d1b55025eea70dc1f30750d2f446084c"
server_key_version: "encryptionkey@0"
I20260619 08:56:11.790941 2472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:11.812564 2472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:11.814404 2472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:11.814512 2472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:11.815006 2472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:11.815047 2472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:11.815070 2472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:11.815081 2472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:11.821282 2472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45917
I20260619 08:56:11.821403 2602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45917 every 8 connection(s)
I20260619 08:56:11.821774 2472 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 08:56:11 dist-test-slave-h4g5 krb5kdc[2075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:11.824834 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:11.815504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44585 (local address 127.0.98.62:43031)
0619 08:56:11.815555 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:11.815557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:11.815790 (+ 233us) server_negotiation.cc:407] Connection header received
0619 08:56:11.816370 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:11.816372 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:11.816385 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:11.816436 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:11.817239 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.818044 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.819545 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.819776 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.819844 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:11.822460 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:11.822462 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:11.822462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:11.822470 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:11.823515 (+ 1045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:11.823891 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:11.823892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:11.823892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:11.823947 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:11.824218 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:11.824218 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:11.824219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:11.824334 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:11.824399 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:11.824661 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 08:56:11.824746 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 08:56:11.825001 2496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:11.815508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43031 (local address 127.0.98.3:44585)
0619 08:56:11.815668 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:11.815704 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:11.816258 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:11.816513 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:11.816518 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:11.816858 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:11.817153 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:11.817160 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.818146 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.818147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:11.819460 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:11.819463 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.819875 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.819876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:11.820042 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:11.820664 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:11.820675 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:11.822371 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:11.823592 (+ 1221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:11.823599 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:11.823601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:11.823824 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:11.824008 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:11.824009 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:11.824009 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:11.824160 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:11.824389 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:11.824395 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:11.824531 (+ 136us) client_negotiation.cc:769] Sending connection context
0619 08:56:11.824610 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 08:56:11.824792 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260619 08:56:11.825418 2603 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43031
I20260619 08:56:11.825500 2603 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:11.825678 2603 heartbeater.cc:511] Master 127.0.98.62:43031 requested a full tablet report, sending...
I20260619 08:56:11.826035 2120 ts_manager.cc:194] Registered new tserver with Master: 1f909d146c124ed7849be361ca427112 (127.0.98.3:45917)
I20260619 08:56:11.826489 2120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44585
I20260619 08:56:11.830061 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 2472
I20260619 08:56:11.830144 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-2/wal/instance
I20260619 08:56:11.830308 392 external_mini_cluster.cc:1428] Setting key b4559e4046e00856872254e76095a5dc
I20260619 08:56:11.831105 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:56:11.831297 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:11.826437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50113 (local address 127.0.98.62:43031)
0619 08:56:11.826476 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:11.826478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:11.826609 (+ 131us) server_negotiation.cc:407] Connection header received
0619 08:56:11.826630 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:11.826631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:11.826640 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:11.826686 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:11.826965 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.827442 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.828121 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.828298 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.828348 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:11.829156 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:11.829158 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:11.829158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:11.829165 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:11.830311 (+ 1146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:11.830640 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:11.830641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:11.830641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:11.830694 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:11.830847 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:11.830848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:11.830848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:11.830949 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:11.831002 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:11.831135 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 08:56:11.831205 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 08:56:11.831311 2609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:11.826377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43031 (local address 127.0.98.3:50113)
0619 08:56:11.826508 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:11.826518 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:11.826577 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:11.826688 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:11.826689 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:11.826849 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:11.826903 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:11.826904 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.827525 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.827526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:11.828021 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:11.828022 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.828352 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.828353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:11.828427 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:11.828808 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:11.828812 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:11.829074 (+ 262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:11.830376 (+ 1302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:11.830377 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:11.830378 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:11.830580 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:11.830744 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:11.830745 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:11.830745 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:11.830800 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:11.831000 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:11.831001 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:11.831043 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 08:56:11.831108 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 08:56:11.831206 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":75,"threads_started":1}
Jun 19 08:56:11 dist-test-slave-h4g5 krb5kdc[2075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 08:56:11 dist-test-slave-h4g5 krb5kdc[2075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:11.852725 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:11.846013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39190 (local address 127.0.98.62:43031)
0619 08:56:11.846079 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:11.846081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:11.846090 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:56:11.846116 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:11.846117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:11.846127 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:11.846174 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:11.846734 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.847216 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.847950 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.848188 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.848240 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:11.850387 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:11.850388 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:11.850389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:11.850396 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:11.851634 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:11.851965 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:11.851966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:11.851966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:11.852019 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:11.852222 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:11.852222 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:11.852223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:11.852350 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:11.852438 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:11.852554 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 08:56:11.852630 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 08:56:11.854509 2120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:56:11.856011 2120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:56:11.862375 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:11.858402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34599 (local address 127.0.0.1:38508)
0619 08:56:11.858516 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:11.858525 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:11.858570 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:11.859136 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:11.859137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:11.859139 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:11.859194 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:11.859196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.860220 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.860221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:11.861285 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:11.861287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.862133 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.862134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:11.862196 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:11.862200 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:56:11.862249 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 08:56:11.862288 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":77,"threads_started":1}
I20260619 08:56:11.862386 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:11.858808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38508 (local address 127.0.98.2:34599)
0619 08:56:11.858974 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:11.858978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:11.858987 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:56:11.859025 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:11.859027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:11.859043 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:11.859256 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:11.859313 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.860055 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.861373 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.862065 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.862119 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:11.862170 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:11.862262 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 08:56:11.862296 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":101,"threads_started":1}
I20260619 08:56:11.862493 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:11.858214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45917 (local address 127.0.0.1:49136)
0619 08:56:11.858324 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:11.858354 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:11.858411 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:11.858699 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:11.858702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:11.858704 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:11.858775 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:11.858777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.860006 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.860007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:11.861119 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:11.861122 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.862006 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.862007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:11.862097 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:11.862115 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 08:56:11.862188 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 08:56:11.862239 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":49,"threads_started":1}
I20260619 08:56:11.862502 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:11.858567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34365 (local address 127.0.0.1:51004)
0619 08:56:11.858900 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:11.858909 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:11.859046 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:11.859292 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:11.859293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:11.859294 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:11.859324 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:11.859326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.860544 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.860545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:11.861525 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:11.861527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.862295 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.862297 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:11.862351 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:11.862354 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:56:11.862400 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 08:56:11.862435 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":65,"threads_started":1}
I20260619 08:56:11.862891 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:11.858684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51004 (local address 127.0.98.1:34365)
0619 08:56:11.858865 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:11.858871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:11.859020 (+ 149us) server_negotiation.cc:407] Connection header received
0619 08:56:11.859136 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:11.859138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:11.859152 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:11.859216 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:11.859399 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.860459 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.861596 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.862130 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.862195 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:11.862263 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:11.862541 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 08:56:11.862590 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":101,"threads_started":1}
I20260619 08:56:11.863070 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:11.858317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49136 (local address 127.0.98.3:45917)
0619 08:56:11.858480 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:11.858485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:11.858494 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:56:11.858529 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:11.858531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:11.858544 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:11.858623 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:11.858857 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.859914 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.861199 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.861922 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.861998 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:11.862058 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:11.862334 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 08:56:11.862376 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20260619 08:56:11.863171 2397 tablet_service.cc:1467] Processing CreateTablet for tablet 45cd23ad1fc74ba9b223c306d48f5205 (DEFAULT_TABLE table=test-table [id=f025200df1594c309dcc4e5e33fb9013]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:11.863569 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45cd23ad1fc74ba9b223c306d48f5205. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:11.863538 2257 tablet_service.cc:1467] Processing CreateTablet for tablet 45cd23ad1fc74ba9b223c306d48f5205 (DEFAULT_TABLE table=test-table [id=f025200df1594c309dcc4e5e33fb9013]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:11.863574 2537 tablet_service.cc:1467] Processing CreateTablet for tablet 45cd23ad1fc74ba9b223c306d48f5205 (DEFAULT_TABLE table=test-table [id=f025200df1594c309dcc4e5e33fb9013]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:11.863855 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45cd23ad1fc74ba9b223c306d48f5205. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:11.863878 2537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45cd23ad1fc74ba9b223c306d48f5205. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:11.866423 2626 tablet_bootstrap.cc:492] T 45cd23ad1fc74ba9b223c306d48f5205 P 1f909d146c124ed7849be361ca427112: Bootstrap starting.
I20260619 08:56:11.867272 2627 tablet_bootstrap.cc:492] T 45cd23ad1fc74ba9b223c306d48f5205 P facf427fe9e541b781955151f2be5909: Bootstrap starting.
I20260619 08:56:11.867344 2626 tablet_bootstrap.cc:654] T 45cd23ad1fc74ba9b223c306d48f5205 P 1f909d146c124ed7849be361ca427112: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:11.867681 2626 log.cc:824] T 45cd23ad1fc74ba9b223c306d48f5205 P 1f909d146c124ed7849be361ca427112: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:11.867893 2627 tablet_bootstrap.cc:654] T 45cd23ad1fc74ba9b223c306d48f5205 P facf427fe9e541b781955151f2be5909: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:11.868233 2628 tablet_bootstrap.cc:492] T 45cd23ad1fc74ba9b223c306d48f5205 P f2fad2d8cc29452d8cef0645665b4a8c: Bootstrap starting.
I20260619 08:56:11.868240 2627 log.cc:824] T 45cd23ad1fc74ba9b223c306d48f5205 P facf427fe9e541b781955151f2be5909: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:11.868791 2628 tablet_bootstrap.cc:654] T 45cd23ad1fc74ba9b223c306d48f5205 P f2fad2d8cc29452d8cef0645665b4a8c: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:11.868938 2627 tablet_bootstrap.cc:492] T 45cd23ad1fc74ba9b223c306d48f5205 P facf427fe9e541b781955151f2be5909: No bootstrap required, opened a new log
I20260619 08:56:11.869007 2627 ts_tablet_manager.cc:1397] T 45cd23ad1fc74ba9b223c306d48f5205 P facf427fe9e541b781955151f2be5909: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:56:11.869108 2628 log.cc:824] T 45cd23ad1fc74ba9b223c306d48f5205 P f2fad2d8cc29452d8cef0645665b4a8c: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:11.869647 2626 tablet_bootstrap.cc:492] T 45cd23ad1fc74ba9b223c306d48f5205 P 1f909d146c124ed7849be361ca427112: No bootstrap required, opened a new log
I20260619 08:56:11.869717 2626 ts_tablet_manager.cc:1397] T 45cd23ad1fc74ba9b223c306d48f5205 P 1f909d146c124ed7849be361ca427112: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 08:56:11.869829 2628 tablet_bootstrap.cc:492] T 45cd23ad1fc74ba9b223c306d48f5205 P f2fad2d8cc29452d8cef0645665b4a8c: No bootstrap required, opened a new log
I20260619 08:56:11.869899 2628 ts_tablet_manager.cc:1397] T 45cd23ad1fc74ba9b223c306d48f5205 P f2fad2d8cc29452d8cef0645665b4a8c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:56:11.870929 2627 raft_consensus.cc:348] T 45cd23ad1fc74ba9b223c306d48f5205 P facf427fe9e541b781955151f2be5909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } }
I20260619 08:56:11.871116 2627 raft_consensus.cc:374] T 45cd23ad1fc74ba9b223c306d48f5205 P facf427fe9e541b781955151f2be5909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:11.871151 2627 raft_consensus.cc:729] T 45cd23ad1fc74ba9b223c306d48f5205 P facf427fe9e541b781955151f2be5909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: facf427fe9e541b781955151f2be5909, State: Initialized, Role: FOLLOWER
I20260619 08:56:11.871301 2627 consensus_queue.cc:260] T 45cd23ad1fc74ba9b223c306d48f5205 P facf427fe9e541b781955151f2be5909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } }
I20260619 08:56:11.871574 2627 ts_tablet_manager.cc:1428] T 45cd23ad1fc74ba9b223c306d48f5205 P facf427fe9e541b781955151f2be5909: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:56:11.871471 2628 raft_consensus.cc:348] T 45cd23ad1fc74ba9b223c306d48f5205 P f2fad2d8cc29452d8cef0645665b4a8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } }
I20260619 08:56:11.871595 2323 heartbeater.cc:503] Master 127.0.98.62:43031 was elected leader, sending a full tablet report...
I20260619 08:56:11.871636 2628 raft_consensus.cc:374] T 45cd23ad1fc74ba9b223c306d48f5205 P f2fad2d8cc29452d8cef0645665b4a8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:11.871670 2628 raft_consensus.cc:729] T 45cd23ad1fc74ba9b223c306d48f5205 P f2fad2d8cc29452d8cef0645665b4a8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fad2d8cc29452d8cef0645665b4a8c, State: Initialized, Role: FOLLOWER
I20260619 08:56:11.871595 2626 raft_consensus.cc:348] T 45cd23ad1fc74ba9b223c306d48f5205 P 1f909d146c124ed7849be361ca427112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } }
I20260619 08:56:11.871722 2626 raft_consensus.cc:374] T 45cd23ad1fc74ba9b223c306d48f5205 P 1f909d146c124ed7849be361ca427112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:11.871744 2626 raft_consensus.cc:729] T 45cd23ad1fc74ba9b223c306d48f5205 P 1f909d146c124ed7849be361ca427112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f909d146c124ed7849be361ca427112, State: Initialized, Role: FOLLOWER
I20260619 08:56:11.871775 2628 consensus_queue.cc:260] T 45cd23ad1fc74ba9b223c306d48f5205 P f2fad2d8cc29452d8cef0645665b4a8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } }
I20260619 08:56:11.871878 2626 consensus_queue.cc:260] T 45cd23ad1fc74ba9b223c306d48f5205 P 1f909d146c124ed7849be361ca427112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } }
I20260619 08:56:11.872005 2628 ts_tablet_manager.cc:1428] T 45cd23ad1fc74ba9b223c306d48f5205 P f2fad2d8cc29452d8cef0645665b4a8c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:56:11.872130 2463 heartbeater.cc:503] Master 127.0.98.62:43031 was elected leader, sending a full tablet report...
I20260619 08:56:11.872143 2626 ts_tablet_manager.cc:1428] T 45cd23ad1fc74ba9b223c306d48f5205 P 1f909d146c124ed7849be361ca427112: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 08:56:11.872252 2603 heartbeater.cc:503] Master 127.0.98.62:43031 was elected leader, sending a full tablet report...
W20260619 08:56:11.910585 2464 tablet.cc:2369] T 45cd23ad1fc74ba9b223c306d48f5205 P f2fad2d8cc29452d8cef0645665b4a8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:56:11.995407 2633 raft_consensus.cc:482] T 45cd23ad1fc74ba9b223c306d48f5205 P f2fad2d8cc29452d8cef0645665b4a8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:56:11.995536 2633 raft_consensus.cc:504] T 45cd23ad1fc74ba9b223c306d48f5205 P f2fad2d8cc29452d8cef0645665b4a8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } }
I20260619 08:56:11.995937 2633 leader_election.cc:288] T 45cd23ad1fc74ba9b223c306d48f5205 P f2fad2d8cc29452d8cef0645665b4a8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f909d146c124ed7849be361ca427112 (127.0.98.3:45917), facf427fe9e541b781955151f2be5909 (127.0.98.1:34365)
I20260619 08:56:11.999723 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:11.996528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48187 (local address 127.0.98.1:34365)
0619 08:56:11.996609 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:11.996611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:11.996618 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:56:11.996642 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:11.996643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:11.996653 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:11.996707 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:11.996847 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.997511 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.998780 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.999356 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.999421 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:11.999432 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:11.999607 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 08:56:11.999645 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 08:56:11.999723 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:11.996229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42737 (local address 127.0.98.3:45917)
0619 08:56:11.996287 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:11.996289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:11.996299 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:56:11.996379 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:11.996380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:11.996390 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:11.996434 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:11.996590 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.997376 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.998776 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.999341 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.999421 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:11.999432 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:11.999611 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 08:56:11.999652 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 08:56:11.999733 2356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:11.996169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45917 (local address 127.0.98.2:42737)
0619 08:56:11.996232 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:11.996243 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:11.996313 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:11.996441 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:11.996442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:11.996443 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:11.996522 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:11.996525 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.997461 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.997462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:11.998671 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:11.998675 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.999443 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.999444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:11.999530 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:11.999536 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 08:56:11.999590 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 08:56:11.999634 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":8}
I20260619 08:56:11.999747 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:11.996256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34365 (local address 127.0.98.2:48187)
0619 08:56:11.996419 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:11.996429 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:11.996481 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:11.996728 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:11.996728 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:11.996730 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:11.996784 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:11.996785 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.997581 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.997582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:11.998701 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:11.998702 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:11.999443 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:11.999444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:11.999530 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:11.999537 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 08:56:11.999590 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 08:56:11.999634 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":4,"thread_start_us":101,"threads_started":1}
I20260619 08:56:12.000156 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd23ad1fc74ba9b223c306d48f5205" candidate_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "facf427fe9e541b781955151f2be5909" is_pre_election: true
I20260619 08:56:12.000164 2557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd23ad1fc74ba9b223c306d48f5205" candidate_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f909d146c124ed7849be361ca427112" is_pre_election: true
I20260619 08:56:12.000322 2277 raft_consensus.cc:2454] T 45cd23ad1fc74ba9b223c306d48f5205 P facf427fe9e541b781955151f2be5909 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2fad2d8cc29452d8cef0645665b4a8c in term 0.
I20260619 08:56:12.000322 2557 raft_consensus.cc:2454] T 45cd23ad1fc74ba9b223c306d48f5205 P 1f909d146c124ed7849be361ca427112 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2fad2d8cc29452d8cef0645665b4a8c in term 0.
I20260619 08:56:12.000546 2351 leader_election.cc:302] T 45cd23ad1fc74ba9b223c306d48f5205 P f2fad2d8cc29452d8cef0645665b4a8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f2fad2d8cc29452d8cef0645665b4a8c, facf427fe9e541b781955151f2be5909; no voters:
I20260619 08:56:12.000676 2633 raft_consensus.cc:2781] T 45cd23ad1fc74ba9b223c306d48f5205 P f2fad2d8cc29452d8cef0645665b4a8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:56:12.000734 2633 raft_consensus.cc:482] T 45cd23ad1fc74ba9b223c306d48f5205 P f2fad2d8cc29452d8cef0645665b4a8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:56:12.000763 2633 raft_consensus.cc:3037] T 45cd23ad1fc74ba9b223c306d48f5205 P f2fad2d8cc29452d8cef0645665b4a8c [term 0 FOLLOWER]: Advancing to term 1
W20260619 08:56:12.001087 2324 tablet.cc:2369] T 45cd23ad1fc74ba9b223c306d48f5205 P facf427fe9e541b781955151f2be5909: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:56:12.001758 2633 raft_consensus.cc:504] T 45cd23ad1fc74ba9b223c306d48f5205 P f2fad2d8cc29452d8cef0645665b4a8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } }
I20260619 08:56:12.001873 2633 leader_election.cc:288] T 45cd23ad1fc74ba9b223c306d48f5205 P f2fad2d8cc29452d8cef0645665b4a8c [CANDIDATE]: Term 1 election: Requested vote from peers 1f909d146c124ed7849be361ca427112 (127.0.98.3:45917), facf427fe9e541b781955151f2be5909 (127.0.98.1:34365)
I20260619 08:56:12.002030 2557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd23ad1fc74ba9b223c306d48f5205" candidate_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f909d146c124ed7849be361ca427112"
I20260619 08:56:12.002113 2557 raft_consensus.cc:3037] T 45cd23ad1fc74ba9b223c306d48f5205 P 1f909d146c124ed7849be361ca427112 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:12.002084 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd23ad1fc74ba9b223c306d48f5205" candidate_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "facf427fe9e541b781955151f2be5909"
I20260619 08:56:12.002169 2277 raft_consensus.cc:3037] T 45cd23ad1fc74ba9b223c306d48f5205 P facf427fe9e541b781955151f2be5909 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:12.002713 2557 raft_consensus.cc:2454] T 45cd23ad1fc74ba9b223c306d48f5205 P 1f909d146c124ed7849be361ca427112 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2fad2d8cc29452d8cef0645665b4a8c in term 1.
I20260619 08:56:12.002871 2351 leader_election.cc:302] T 45cd23ad1fc74ba9b223c306d48f5205 P f2fad2d8cc29452d8cef0645665b4a8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f909d146c124ed7849be361ca427112, f2fad2d8cc29452d8cef0645665b4a8c; no voters:
I20260619 08:56:12.002907 2277 raft_consensus.cc:2454] T 45cd23ad1fc74ba9b223c306d48f5205 P facf427fe9e541b781955151f2be5909 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2fad2d8cc29452d8cef0645665b4a8c in term 1.
I20260619 08:56:12.002962 2633 raft_consensus.cc:2781] T 45cd23ad1fc74ba9b223c306d48f5205 P f2fad2d8cc29452d8cef0645665b4a8c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:12.003123 2633 raft_consensus.cc:686] T 45cd23ad1fc74ba9b223c306d48f5205 P f2fad2d8cc29452d8cef0645665b4a8c [term 1 LEADER]: Becoming Leader. State: Replica: f2fad2d8cc29452d8cef0645665b4a8c, State: Running, Role: LEADER
I20260619 08:56:12.003249 2633 consensus_queue.cc:237] T 45cd23ad1fc74ba9b223c306d48f5205 P f2fad2d8cc29452d8cef0645665b4a8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } }
I20260619 08:56:12.004065 2119 catalog_manager.cc:5495] T 45cd23ad1fc74ba9b223c306d48f5205 P f2fad2d8cc29452d8cef0645665b4a8c reported cstate change: term changed from 0 to 1, leader changed from <none> to f2fad2d8cc29452d8cef0645665b4a8c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } health_report { overall_health: UNKNOWN } } }
Jun 19 08:56:12 dist-test-slave-h4g5 krb5kdc[2075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:12.035393 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:12.028895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39198 (local address 127.0.98.62:43031)
0619 08:56:12.028986 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:12.028989 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:12.029101 (+ 112us) server_negotiation.cc:407] Connection header received
0619 08:56:12.029184 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:12.029184 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:12.029194 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:12.029244 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:12.029699 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:12.030179 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:12.030824 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:12.030998 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:12.031050 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:12.033075 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:12.033077 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:12.033077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:12.033084 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:12.034311 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:12.034650 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:12.034651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:12.034652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:12.034711 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:12.034860 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:12.034861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:12.034861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:12.034967 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:12.035019 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:12.035180 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 08:56:12.035314 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":54}
I20260619 08:56:12.035392 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:12.028816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43031 (local address 127.0.0.1:39198)
0619 08:56:12.029029 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:12.029040 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:12.029129 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:12.029254 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:12.029256 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:12.029568 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:12.029620 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:12.029624 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:12.030246 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:12.030247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:12.030756 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:12.030757 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:12.031052 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:12.031053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:12.031105 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:12.031584 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:12.031589 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:12.032993 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:12.034390 (+ 1397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:12.034397 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:12.034400 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:12.034578 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:12.034762 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:12.034763 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:12.034763 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:12.034814 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:12.035016 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:12.035044 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:12.035099 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 08:56:12.035153 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 08:56:12.035269 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":100,"threads_started":1}
I20260619 08:56:12.037415 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39198:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 08:56:12.037619 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:56:12.038971 2257 tablet_service.cc:1467] Processing CreateTablet for tablet 48bc406891534ad0a72e2f87ccfde4a8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=815fa1e6d2494137bf94eb04a7ae7f31]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 08:56:12.039018 2397 tablet_service.cc:1467] Processing CreateTablet for tablet 48bc406891534ad0a72e2f87ccfde4a8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=815fa1e6d2494137bf94eb04a7ae7f31]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 08:56:12.039067 2537 tablet_service.cc:1467] Processing CreateTablet for tablet 48bc406891534ad0a72e2f87ccfde4a8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=815fa1e6d2494137bf94eb04a7ae7f31]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 08:56:12.039151 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48bc406891534ad0a72e2f87ccfde4a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:12.039191 2537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48bc406891534ad0a72e2f87ccfde4a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:12.039192 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48bc406891534ad0a72e2f87ccfde4a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:12.040302 2627 tablet_bootstrap.cc:492] T 48bc406891534ad0a72e2f87ccfde4a8 P facf427fe9e541b781955151f2be5909: Bootstrap starting.
I20260619 08:56:12.040614 2628 tablet_bootstrap.cc:492] T 48bc406891534ad0a72e2f87ccfde4a8 P f2fad2d8cc29452d8cef0645665b4a8c: Bootstrap starting.
I20260619 08:56:12.040614 2626 tablet_bootstrap.cc:492] T 48bc406891534ad0a72e2f87ccfde4a8 P 1f909d146c124ed7849be361ca427112: Bootstrap starting.
I20260619 08:56:12.040683 2627 tablet_bootstrap.cc:654] T 48bc406891534ad0a72e2f87ccfde4a8 P facf427fe9e541b781955151f2be5909: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:12.041016 2628 tablet_bootstrap.cc:654] T 48bc406891534ad0a72e2f87ccfde4a8 P f2fad2d8cc29452d8cef0645665b4a8c: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:12.041114 2626 tablet_bootstrap.cc:654] T 48bc406891534ad0a72e2f87ccfde4a8 P 1f909d146c124ed7849be361ca427112: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:12.041702 2628 tablet_bootstrap.cc:492] T 48bc406891534ad0a72e2f87ccfde4a8 P f2fad2d8cc29452d8cef0645665b4a8c: No bootstrap required, opened a new log
I20260619 08:56:12.041743 2628 ts_tablet_manager.cc:1397] T 48bc406891534ad0a72e2f87ccfde4a8 P f2fad2d8cc29452d8cef0645665b4a8c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 08:56:12.041752 2627 tablet_bootstrap.cc:492] T 48bc406891534ad0a72e2f87ccfde4a8 P facf427fe9e541b781955151f2be5909: No bootstrap required, opened a new log
I20260619 08:56:12.041795 2627 ts_tablet_manager.cc:1397] T 48bc406891534ad0a72e2f87ccfde4a8 P facf427fe9e541b781955151f2be5909: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:56:12.041882 2628 raft_consensus.cc:348] T 48bc406891534ad0a72e2f87ccfde4a8 P f2fad2d8cc29452d8cef0645665b4a8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:12.041957 2628 raft_consensus.cc:374] T 48bc406891534ad0a72e2f87ccfde4a8 P f2fad2d8cc29452d8cef0645665b4a8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:12.041973 2628 raft_consensus.cc:729] T 48bc406891534ad0a72e2f87ccfde4a8 P f2fad2d8cc29452d8cef0645665b4a8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fad2d8cc29452d8cef0645665b4a8c, State: Initialized, Role: FOLLOWER
I20260619 08:56:12.041934 2627 raft_consensus.cc:348] T 48bc406891534ad0a72e2f87ccfde4a8 P facf427fe9e541b781955151f2be5909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:12.041993 2627 raft_consensus.cc:374] T 48bc406891534ad0a72e2f87ccfde4a8 P facf427fe9e541b781955151f2be5909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:12.042018 2627 raft_consensus.cc:729] T 48bc406891534ad0a72e2f87ccfde4a8 P facf427fe9e541b781955151f2be5909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: facf427fe9e541b781955151f2be5909, State: Initialized, Role: FOLLOWER
I20260619 08:56:12.042011 2628 consensus_queue.cc:260] T 48bc406891534ad0a72e2f87ccfde4a8 P f2fad2d8cc29452d8cef0645665b4a8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:12.042068 2627 consensus_queue.cc:260] T 48bc406891534ad0a72e2f87ccfde4a8 P facf427fe9e541b781955151f2be5909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:12.042086 2628 ts_tablet_manager.cc:1428] T 48bc406891534ad0a72e2f87ccfde4a8 P f2fad2d8cc29452d8cef0645665b4a8c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:12.042162 2627 ts_tablet_manager.cc:1428] T 48bc406891534ad0a72e2f87ccfde4a8 P facf427fe9e541b781955151f2be5909: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:12.042207 2633 tablet_replica.cc:439] T 48bc406891534ad0a72e2f87ccfde4a8 P f2fad2d8cc29452d8cef0645665b4a8c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } }
I20260619 08:56:12.042268 2633 tablet_replica.cc:442] T 48bc406891534ad0a72e2f87ccfde4a8 P f2fad2d8cc29452d8cef0645665b4a8c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:56:12.042384 2632 tablet_replica.cc:439] T 48bc406891534ad0a72e2f87ccfde4a8 P facf427fe9e541b781955151f2be5909: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } }
I20260619 08:56:12.042464 2632 tablet_replica.cc:442] T 48bc406891534ad0a72e2f87ccfde4a8 P facf427fe9e541b781955151f2be5909: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:56:12.042680 2626 tablet_bootstrap.cc:492] T 48bc406891534ad0a72e2f87ccfde4a8 P 1f909d146c124ed7849be361ca427112: No bootstrap required, opened a new log
I20260619 08:56:12.042732 2626 ts_tablet_manager.cc:1397] T 48bc406891534ad0a72e2f87ccfde4a8 P 1f909d146c124ed7849be361ca427112: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:56:12.042877 2626 raft_consensus.cc:348] T 48bc406891534ad0a72e2f87ccfde4a8 P 1f909d146c124ed7849be361ca427112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:12.042949 2626 raft_consensus.cc:374] T 48bc406891534ad0a72e2f87ccfde4a8 P 1f909d146c124ed7849be361ca427112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:12.042975 2626 raft_consensus.cc:729] T 48bc406891534ad0a72e2f87ccfde4a8 P 1f909d146c124ed7849be361ca427112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f909d146c124ed7849be361ca427112, State: Initialized, Role: FOLLOWER
I20260619 08:56:12.043021 2626 consensus_queue.cc:260] T 48bc406891534ad0a72e2f87ccfde4a8 P 1f909d146c124ed7849be361ca427112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:12.043112 2626 ts_tablet_manager.cc:1428] T 48bc406891534ad0a72e2f87ccfde4a8 P 1f909d146c124ed7849be361ca427112: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:12.043195 2634 tablet_replica.cc:439] T 48bc406891534ad0a72e2f87ccfde4a8 P 1f909d146c124ed7849be361ca427112: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } }
I20260619 08:56:12.043292 2634 tablet_replica.cc:442] T 48bc406891534ad0a72e2f87ccfde4a8 P 1f909d146c124ed7849be361ca427112: This TxnStatusTablet replica's current role is: FOLLOWER
W20260619 08:56:12.072446 2604 tablet.cc:2369] T 45cd23ad1fc74ba9b223c306d48f5205 P 1f909d146c124ed7849be361ca427112: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:56:12.180030 2632 raft_consensus.cc:482] T 48bc406891534ad0a72e2f87ccfde4a8 P facf427fe9e541b781955151f2be5909 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:56:12.180154 2632 raft_consensus.cc:504] T 48bc406891534ad0a72e2f87ccfde4a8 P facf427fe9e541b781955151f2be5909 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:12.180462 2632 leader_election.cc:288] T 48bc406891534ad0a72e2f87ccfde4a8 P facf427fe9e541b781955151f2be5909 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2fad2d8cc29452d8cef0645665b4a8c (127.0.98.2:34599), 1f909d146c124ed7849be361ca427112 (127.0.98.3:45917)
I20260619 08:56:12.183758 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:12.180723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33107 (local address 127.0.98.2:34599)
0619 08:56:12.180782 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:12.180784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:12.180852 (+ 68us) server_negotiation.cc:407] Connection header received
0619 08:56:12.180943 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:12.180945 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:12.180957 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:12.180988 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:12.181223 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:12.181785 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:12.182858 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:12.183414 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:12.183497 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:12.183509 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:12.183665 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 08:56:12.183699 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 08:56:12.183833 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:12.180642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34599 (local address 127.0.98.1:33107)
0619 08:56:12.180758 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:12.180770 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:12.180857 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:12.181057 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:12.181058 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:12.181060 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:12.181143 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:12.181145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:12.181883 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:12.181884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:12.182770 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:12.182773 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:12.183523 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:12.183524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:12.183591 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:12.183596 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 08:56:12.183695 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 08:56:12.183736 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":50,"thread_start_us":63,"threads_started":1}
I20260619 08:56:12.184217 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48bc406891534ad0a72e2f87ccfde4a8" candidate_uuid: "facf427fe9e541b781955151f2be5909" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" is_pre_election: true
I20260619 08:56:12.184345 2417 raft_consensus.cc:2454] T 48bc406891534ad0a72e2f87ccfde4a8 P f2fad2d8cc29452d8cef0645665b4a8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate facf427fe9e541b781955151f2be5909 in term 0.
I20260619 08:56:12.184379 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:12.180934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49619 (local address 127.0.98.3:45917)
0619 08:56:12.181001 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:12.181003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:12.181012 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:56:12.181059 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:12.181060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:12.181067 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:12.181118 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:12.181317 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:12.181782 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:12.183538 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:12.184043 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:12.184122 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:12.184132 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:12.184280 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 08:56:12.184320 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 08:56:12.184386 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:12.180812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45917 (local address 127.0.98.1:49619)
0619 08:56:12.180910 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:12.180922 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:12.180996 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:12.181132 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:12.181132 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:12.181134 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:12.181211 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:12.181213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:12.181874 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:12.181875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:12.183457 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:12.183460 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:12.184146 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:12.184147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:12.184211 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:12.184215 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:56:12.184267 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 08:56:12.184302 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":28,"thread_start_us":54,"threads_started":1}
I20260619 08:56:12.184568 2211 leader_election.cc:302] T 48bc406891534ad0a72e2f87ccfde4a8 P facf427fe9e541b781955151f2be5909 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f2fad2d8cc29452d8cef0645665b4a8c, facf427fe9e541b781955151f2be5909; no voters:
I20260619 08:56:12.184772 2632 raft_consensus.cc:2781] T 48bc406891534ad0a72e2f87ccfde4a8 P facf427fe9e541b781955151f2be5909 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:56:12.184808 2632 raft_consensus.cc:482] T 48bc406891534ad0a72e2f87ccfde4a8 P facf427fe9e541b781955151f2be5909 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:56:12.184827 2632 raft_consensus.cc:3037] T 48bc406891534ad0a72e2f87ccfde4a8 P facf427fe9e541b781955151f2be5909 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:12.184840 2557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48bc406891534ad0a72e2f87ccfde4a8" candidate_uuid: "facf427fe9e541b781955151f2be5909" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f909d146c124ed7849be361ca427112" is_pre_election: true
I20260619 08:56:12.184911 2557 raft_consensus.cc:2454] T 48bc406891534ad0a72e2f87ccfde4a8 P 1f909d146c124ed7849be361ca427112 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate facf427fe9e541b781955151f2be5909 in term 0.
I20260619 08:56:12.185577 2632 raft_consensus.cc:504] T 48bc406891534ad0a72e2f87ccfde4a8 P facf427fe9e541b781955151f2be5909 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:12.185717 2632 leader_election.cc:288] T 48bc406891534ad0a72e2f87ccfde4a8 P facf427fe9e541b781955151f2be5909 [CANDIDATE]: Term 1 election: Requested vote from peers f2fad2d8cc29452d8cef0645665b4a8c (127.0.98.2:34599), 1f909d146c124ed7849be361ca427112 (127.0.98.3:45917)
I20260619 08:56:12.185933 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48bc406891534ad0a72e2f87ccfde4a8" candidate_uuid: "facf427fe9e541b781955151f2be5909" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fad2d8cc29452d8cef0645665b4a8c"
I20260619 08:56:12.186012 2417 raft_consensus.cc:3037] T 48bc406891534ad0a72e2f87ccfde4a8 P f2fad2d8cc29452d8cef0645665b4a8c [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:12.186635 2557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48bc406891534ad0a72e2f87ccfde4a8" candidate_uuid: "facf427fe9e541b781955151f2be5909" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f909d146c124ed7849be361ca427112"
I20260619 08:56:12.186723 2557 raft_consensus.cc:3037] T 48bc406891534ad0a72e2f87ccfde4a8 P 1f909d146c124ed7849be361ca427112 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:12.186810 2417 raft_consensus.cc:2454] T 48bc406891534ad0a72e2f87ccfde4a8 P f2fad2d8cc29452d8cef0645665b4a8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate facf427fe9e541b781955151f2be5909 in term 1.
I20260619 08:56:12.186956 2211 leader_election.cc:302] T 48bc406891534ad0a72e2f87ccfde4a8 P facf427fe9e541b781955151f2be5909 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f2fad2d8cc29452d8cef0645665b4a8c, facf427fe9e541b781955151f2be5909; no voters:
I20260619 08:56:12.187073 2632 raft_consensus.cc:2781] T 48bc406891534ad0a72e2f87ccfde4a8 P facf427fe9e541b781955151f2be5909 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:12.187201 2632 raft_consensus.cc:686] T 48bc406891534ad0a72e2f87ccfde4a8 P facf427fe9e541b781955151f2be5909 [term 1 LEADER]: Becoming Leader. State: Replica: facf427fe9e541b781955151f2be5909, State: Running, Role: LEADER
I20260619 08:56:12.187323 2632 consensus_queue.cc:237] T 48bc406891534ad0a72e2f87ccfde4a8 P facf427fe9e541b781955151f2be5909 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:12.187506 2557 raft_consensus.cc:2454] T 48bc406891534ad0a72e2f87ccfde4a8 P 1f909d146c124ed7849be361ca427112 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate facf427fe9e541b781955151f2be5909 in term 1.
I20260619 08:56:12.187799 2649 tablet_replica.cc:439] T 48bc406891534ad0a72e2f87ccfde4a8 P facf427fe9e541b781955151f2be5909: TxnStatusTablet state changed. Reason: New leader facf427fe9e541b781955151f2be5909. Latest consensus state: current_term: 1 leader_uuid: "facf427fe9e541b781955151f2be5909" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } }
I20260619 08:56:12.187906 2649 tablet_replica.cc:442] T 48bc406891534ad0a72e2f87ccfde4a8 P facf427fe9e541b781955151f2be5909: This TxnStatusTablet replica's current role is: LEADER
I20260619 08:56:12.188162 2651 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 08:56:12.188181 2120 catalog_manager.cc:5495] T 48bc406891534ad0a72e2f87ccfde4a8 P facf427fe9e541b781955151f2be5909 reported cstate change: term changed from 0 to 1, leader changed from <none> to facf427fe9e541b781955151f2be5909 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "facf427fe9e541b781955151f2be5909" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } health_report { overall_health: UNKNOWN } } }
I20260619 08:56:12.188287 2651 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 08:56:12.225895 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:12.223136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51008 (local address 127.0.98.1:34365)
0619 08:56:12.223190 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:12.223192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:12.223201 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:56:12.223283 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:12.223284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:12.223300 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:12.223355 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:56:12.223497 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:12.223957 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:12.225082 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:12.225285 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:12.225339 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:12.225538 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:56:12.225639 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:56:12.225702 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:12.225760 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 08:56:12.225780 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 08:56:12.225894 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:12.223034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34365 (local address 127.0.0.1:51008)
0619 08:56:12.223127 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:12.223141 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:12.223197 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:12.223375 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:12.223376 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:12.223378 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 08:56:12.223434 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:12.223437 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:12.224026 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:12.224027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:12.225001 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:12.225005 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:12.225353 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:12.225354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:12.225420 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:12.225439 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 08:56:12.225699 (+ 260us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 08:56:12.225705 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 08:56:12.225756 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 08:56:12.225807 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 08:56:12.227944 2557 raft_consensus.cc:1261] T 48bc406891534ad0a72e2f87ccfde4a8 P 1f909d146c124ed7849be361ca427112 [term 1 FOLLOWER]: Refusing update from remote peer facf427fe9e541b781955151f2be5909: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:56:12.227944 2417 raft_consensus.cc:1261] T 48bc406891534ad0a72e2f87ccfde4a8 P f2fad2d8cc29452d8cef0645665b4a8c [term 1 FOLLOWER]: Refusing update from remote peer facf427fe9e541b781955151f2be5909: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:56:12.228263 2649 consensus_queue.cc:1035] T 48bc406891534ad0a72e2f87ccfde4a8 P facf427fe9e541b781955151f2be5909 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:56:12.228433 2632 consensus_queue.cc:1035] T 48bc406891534ad0a72e2f87ccfde4a8 P facf427fe9e541b781955151f2be5909 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:56:12.229570 2633 tablet_replica.cc:439] T 48bc406891534ad0a72e2f87ccfde4a8 P f2fad2d8cc29452d8cef0645665b4a8c: TxnStatusTablet state changed. Reason: New leader facf427fe9e541b781955151f2be5909. Latest consensus state: current_term: 1 leader_uuid: "facf427fe9e541b781955151f2be5909" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } }
I20260619 08:56:12.229674 2633 tablet_replica.cc:442] T 48bc406891534ad0a72e2f87ccfde4a8 P f2fad2d8cc29452d8cef0645665b4a8c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:56:12.229843 2634 tablet_replica.cc:439] T 48bc406891534ad0a72e2f87ccfde4a8 P 1f909d146c124ed7849be361ca427112: TxnStatusTablet state changed. Reason: New leader facf427fe9e541b781955151f2be5909. Latest consensus state: current_term: 1 leader_uuid: "facf427fe9e541b781955151f2be5909" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } }
I20260619 08:56:12.229934 2634 tablet_replica.cc:442] T 48bc406891534ad0a72e2f87ccfde4a8 P 1f909d146c124ed7849be361ca427112: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:56:12.230711 2632 tablet_replica.cc:439] T 48bc406891534ad0a72e2f87ccfde4a8 P facf427fe9e541b781955151f2be5909: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "facf427fe9e541b781955151f2be5909" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } }
I20260619 08:56:12.230799 2632 tablet_replica.cc:442] T 48bc406891534ad0a72e2f87ccfde4a8 P facf427fe9e541b781955151f2be5909: This TxnStatusTablet replica's current role is: LEADER
I20260619 08:56:12.231272 2656 mvcc.cc:205] Tried to move back new op lower bound from 7298495988642496512 to 7298495988479746048. Current Snapshot: MvccSnapshot[applied={T|T < 7298495988642496512}]
I20260619 08:56:12.231367 2634 tablet_replica.cc:439] T 48bc406891534ad0a72e2f87ccfde4a8 P 1f909d146c124ed7849be361ca427112: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "facf427fe9e541b781955151f2be5909" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } }
I20260619 08:56:12.231431 2634 tablet_replica.cc:442] T 48bc406891534ad0a72e2f87ccfde4a8 P 1f909d146c124ed7849be361ca427112: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:56:12.232394 2649 tablet_replica.cc:439] T 48bc406891534ad0a72e2f87ccfde4a8 P facf427fe9e541b781955151f2be5909: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "facf427fe9e541b781955151f2be5909" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } }
I20260619 08:56:12.232474 2649 tablet_replica.cc:442] T 48bc406891534ad0a72e2f87ccfde4a8 P facf427fe9e541b781955151f2be5909: This TxnStatusTablet replica's current role is: LEADER
I20260619 08:56:12.232795 2655 mvcc.cc:205] Tried to move back new op lower bound from 7298495988642496512 to 7298495988479746048. Current Snapshot: MvccSnapshot[applied={T|T < 7298495988642496512}]
I20260619 08:56:12.232892 2633 tablet_replica.cc:439] T 48bc406891534ad0a72e2f87ccfde4a8 P f2fad2d8cc29452d8cef0645665b4a8c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "facf427fe9e541b781955151f2be5909" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } }
I20260619 08:56:12.232952 2633 tablet_replica.cc:442] T 48bc406891534ad0a72e2f87ccfde4a8 P f2fad2d8cc29452d8cef0645665b4a8c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:56:12.235471 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:12.232437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38512 (local address 127.0.98.2:34599)
0619 08:56:12.232506 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:12.232508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:12.232515 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:56:12.232561 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:12.232561 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:12.232580 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:12.232647 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:56:12.233163 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:12.233902 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:12.234697 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:12.235017 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:12.235081 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:12.235180 (+ 99us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:56:12.235294 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:56:12.235361 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:12.235377 (+ 16us) server_negotiation.cc:299] Negotiation successful
0619 08:56:12.235399 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 08:56:12.240548 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:12.237923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47695 (local address 127.0.98.1:34365)
0619 08:56:12.237969 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:12.237971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:12.238064 (+ 93us) server_negotiation.cc:407] Connection header received
0619 08:56:12.238131 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:12.238131 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:12.238142 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:12.238184 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:56:12.238305 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:12.238766 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:12.239839 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:12.240023 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:12.240103 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:12.240315 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:56:12.240361 (+ 46us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:56:12.240412 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:12.240474 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 08:56:12.240492 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 08:56:12.240604 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:12.237853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34365 (local address 127.0.98.2:47695)
0619 08:56:12.237996 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:12.238008 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:12.238067 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:12.238194 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:12.238195 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:12.238196 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 08:56:12.238255 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:12.238257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:12.238829 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:12.238830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:12.239752 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:12.239755 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:12.240119 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:12.240120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:12.240181 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:12.240198 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 08:56:12.240422 (+ 224us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 08:56:12.240424 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:56:12.240471 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 08:56:12.240511 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
I20260619 08:56:12.243183 2557 raft_consensus.cc:1261] T 45cd23ad1fc74ba9b223c306d48f5205 P 1f909d146c124ed7849be361ca427112 [term 1 FOLLOWER]: Refusing update from remote peer f2fad2d8cc29452d8cef0645665b4a8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:56:12.243198 2277 raft_consensus.cc:1261] T 45cd23ad1fc74ba9b223c306d48f5205 P facf427fe9e541b781955151f2be5909 [term 1 FOLLOWER]: Refusing update from remote peer f2fad2d8cc29452d8cef0645665b4a8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:56:12.243435 2636 consensus_queue.cc:1035] T 45cd23ad1fc74ba9b223c306d48f5205 P f2fad2d8cc29452d8cef0645665b4a8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:56:12.243573 2633 consensus_queue.cc:1035] T 45cd23ad1fc74ba9b223c306d48f5205 P f2fad2d8cc29452d8cef0645665b4a8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:56:12.245314 2654 mvcc.cc:205] Tried to move back new op lower bound from 7298495988706205696 to 7298495987725701120. Current Snapshot: MvccSnapshot[applied={T|T < 7298495988706205696}]
I20260619 08:56:12.253018 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:12.249950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34599 (local address 127.0.98.1:37119)
0619 08:56:12.250222 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:12.250232 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:12.250300 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:12.250447 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:12.250448 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:12.250450 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 08:56:12.250491 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:12.250492 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:12.251344 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:12.251345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:12.252223 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:12.252226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:12.252666 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:12.252667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:12.252718 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:12.252731 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 08:56:12.252890 (+ 159us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 08:56:12.252891 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 08:56:12.252920 (+ 29us) client_negotiation.cc:240] Negotiation successful
0619 08:56:12.252957 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":82,"threads_started":1}
I20260619 08:56:12.253021 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:12.250138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37119 (local address 127.0.98.2:34599)
0619 08:56:12.250179 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:12.250181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:12.250275 (+ 94us) server_negotiation.cc:407] Connection header received
0619 08:56:12.250377 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:12.250378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:12.250392 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:12.250445 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:56:12.250532 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:12.251245 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:12.252297 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:12.252579 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:12.252650 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:12.252800 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:56:12.252834 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:56:12.252888 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:12.252926 (+ 38us) server_negotiation.cc:299] Negotiation successful
0619 08:56:12.252946 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 08:56:12.262415 2118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:56:12.262548 2118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:56:12.263337 2118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5efeda2ae7b6428aada15b2780867507: Sending DeleteTablet for 3 replicas of tablet 45cd23ad1fc74ba9b223c306d48f5205
I20260619 08:56:12.263847 2257 tablet_service.cc:1514] Processing DeleteTablet for tablet 45cd23ad1fc74ba9b223c306d48f5205 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51004
I20260619 08:56:12.263846 2537 tablet_service.cc:1514] Processing DeleteTablet for tablet 45cd23ad1fc74ba9b223c306d48f5205 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49136
I20260619 08:56:12.264138 2397 tablet_service.cc:1514] Processing DeleteTablet for tablet 45cd23ad1fc74ba9b223c306d48f5205 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38508
I20260619 08:56:12.264178 2675 tablet_replica.cc:330] T 45cd23ad1fc74ba9b223c306d48f5205 P facf427fe9e541b781955151f2be5909: stopping tablet replica
I20260619 08:56:12.264284 2675 raft_consensus.cc:2229] T 45cd23ad1fc74ba9b223c306d48f5205 P facf427fe9e541b781955151f2be5909 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:56:12.264338 2675 raft_consensus.cc:2258] T 45cd23ad1fc74ba9b223c306d48f5205 P facf427fe9e541b781955151f2be5909 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:12.264777 2675 ts_tablet_manager.cc:1905] T 45cd23ad1fc74ba9b223c306d48f5205 P facf427fe9e541b781955151f2be5909: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:12.264761 2118 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:56:12.264899 2118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:56:12.266526 2257 tablet_service.cc:1467] Processing CreateTablet for tablet 0859ba6f857d4263a1bd5baf244d83da (DEFAULT_TABLE table=test-table [id=898cb918835d404eb62cc61d258cf604]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:12.266567 2675 ts_tablet_manager.cc:1918] T 45cd23ad1fc74ba9b223c306d48f5205 P facf427fe9e541b781955151f2be5909: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 08:56:12.266631 2675 log.cc:1196] T 45cd23ad1fc74ba9b223c306d48f5205 P facf427fe9e541b781955151f2be5909: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-0/wal/wals/45cd23ad1fc74ba9b223c306d48f5205
I20260619 08:56:12.266646 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0859ba6f857d4263a1bd5baf244d83da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:12.267184 2675 ts_tablet_manager.cc:1939] T 45cd23ad1fc74ba9b223c306d48f5205 P facf427fe9e541b781955151f2be5909: Deleting consensus metadata
I20260619 08:56:12.267926 2627 tablet_bootstrap.cc:492] T 0859ba6f857d4263a1bd5baf244d83da P facf427fe9e541b781955151f2be5909: Bootstrap starting.
I20260619 08:56:12.268035 2537 tablet_service.cc:1467] Processing CreateTablet for tablet 0859ba6f857d4263a1bd5baf244d83da (DEFAULT_TABLE table=test-table [id=898cb918835d404eb62cc61d258cf604]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:12.268169 2537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0859ba6f857d4263a1bd5baf244d83da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:12.268343 2627 tablet_bootstrap.cc:654] T 0859ba6f857d4263a1bd5baf244d83da P facf427fe9e541b781955151f2be5909: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:12.268896 2627 tablet_bootstrap.cc:492] T 0859ba6f857d4263a1bd5baf244d83da P facf427fe9e541b781955151f2be5909: No bootstrap required, opened a new log
I20260619 08:56:12.268939 2627 ts_tablet_manager.cc:1397] T 0859ba6f857d4263a1bd5baf244d83da P facf427fe9e541b781955151f2be5909: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 08:56:12.269071 2627 raft_consensus.cc:348] T 0859ba6f857d4263a1bd5baf244d83da P facf427fe9e541b781955151f2be5909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:12.269150 2627 raft_consensus.cc:374] T 0859ba6f857d4263a1bd5baf244d83da P facf427fe9e541b781955151f2be5909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:12.269170 2627 raft_consensus.cc:729] T 0859ba6f857d4263a1bd5baf244d83da P facf427fe9e541b781955151f2be5909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: facf427fe9e541b781955151f2be5909, State: Initialized, Role: FOLLOWER
I20260619 08:56:12.269212 2627 consensus_queue.cc:260] T 0859ba6f857d4263a1bd5baf244d83da P facf427fe9e541b781955151f2be5909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:12.269301 2627 ts_tablet_manager.cc:1428] T 0859ba6f857d4263a1bd5baf244d83da P facf427fe9e541b781955151f2be5909: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:12.269444 2626 tablet_bootstrap.cc:492] T 0859ba6f857d4263a1bd5baf244d83da P 1f909d146c124ed7849be361ca427112: Bootstrap starting.
I20260619 08:56:12.269812 2626 tablet_bootstrap.cc:654] T 0859ba6f857d4263a1bd5baf244d83da P 1f909d146c124ed7849be361ca427112: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:12.270300 2397 tablet_service.cc:1467] Processing CreateTablet for tablet 0859ba6f857d4263a1bd5baf244d83da (DEFAULT_TABLE table=test-table [id=898cb918835d404eb62cc61d258cf604]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:12.270416 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0859ba6f857d4263a1bd5baf244d83da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:12.270450 2626 tablet_bootstrap.cc:492] T 0859ba6f857d4263a1bd5baf244d83da P 1f909d146c124ed7849be361ca427112: No bootstrap required, opened a new log
I20260619 08:56:12.270493 2626 ts_tablet_manager.cc:1397] T 0859ba6f857d4263a1bd5baf244d83da P 1f909d146c124ed7849be361ca427112: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:12.270627 2626 raft_consensus.cc:348] T 0859ba6f857d4263a1bd5baf244d83da P 1f909d146c124ed7849be361ca427112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:12.270691 2626 raft_consensus.cc:374] T 0859ba6f857d4263a1bd5baf244d83da P 1f909d146c124ed7849be361ca427112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:12.270711 2626 raft_consensus.cc:729] T 0859ba6f857d4263a1bd5baf244d83da P 1f909d146c124ed7849be361ca427112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f909d146c124ed7849be361ca427112, State: Initialized, Role: FOLLOWER
I20260619 08:56:12.270753 2626 consensus_queue.cc:260] T 0859ba6f857d4263a1bd5baf244d83da P 1f909d146c124ed7849be361ca427112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:12.270836 2626 ts_tablet_manager.cc:1428] T 0859ba6f857d4263a1bd5baf244d83da P 1f909d146c124ed7849be361ca427112: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:12.272744 2107 catalog_manager.cc:4841] TS facf427fe9e541b781955151f2be5909 (127.0.98.1:34365): tablet 45cd23ad1fc74ba9b223c306d48f5205 (table test-table [id=f025200df1594c309dcc4e5e33fb9013]) successfully deleted
I20260619 08:56:12.273394 2628 tablet_bootstrap.cc:492] T 0859ba6f857d4263a1bd5baf244d83da P f2fad2d8cc29452d8cef0645665b4a8c: Bootstrap starting.
I20260619 08:56:12.273435 2677 tablet_replica.cc:330] T 45cd23ad1fc74ba9b223c306d48f5205 P f2fad2d8cc29452d8cef0645665b4a8c: stopping tablet replica
I20260619 08:56:12.273732 2677 raft_consensus.cc:2229] T 45cd23ad1fc74ba9b223c306d48f5205 P f2fad2d8cc29452d8cef0645665b4a8c [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:56:12.273810 2628 tablet_bootstrap.cc:654] T 0859ba6f857d4263a1bd5baf244d83da P f2fad2d8cc29452d8cef0645665b4a8c: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:12.273952 2677 raft_consensus.cc:2258] T 45cd23ad1fc74ba9b223c306d48f5205 P f2fad2d8cc29452d8cef0645665b4a8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:12.274171 2676 tablet_replica.cc:330] T 45cd23ad1fc74ba9b223c306d48f5205 P 1f909d146c124ed7849be361ca427112: stopping tablet replica
I20260619 08:56:12.274325 2676 raft_consensus.cc:2229] T 45cd23ad1fc74ba9b223c306d48f5205 P 1f909d146c124ed7849be361ca427112 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:56:12.274427 2676 raft_consensus.cc:2258] T 45cd23ad1fc74ba9b223c306d48f5205 P 1f909d146c124ed7849be361ca427112 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:12.275045 2676 ts_tablet_manager.cc:1905] T 45cd23ad1fc74ba9b223c306d48f5205 P 1f909d146c124ed7849be361ca427112: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:12.275084 2677 ts_tablet_manager.cc:1905] T 45cd23ad1fc74ba9b223c306d48f5205 P f2fad2d8cc29452d8cef0645665b4a8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:12.276767 2676 ts_tablet_manager.cc:1918] T 45cd23ad1fc74ba9b223c306d48f5205 P 1f909d146c124ed7849be361ca427112: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 08:56:12.276887 2676 log.cc:1196] T 45cd23ad1fc74ba9b223c306d48f5205 P 1f909d146c124ed7849be361ca427112: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-2/wal/wals/45cd23ad1fc74ba9b223c306d48f5205
I20260619 08:56:12.277002 2677 ts_tablet_manager.cc:1918] T 45cd23ad1fc74ba9b223c306d48f5205 P f2fad2d8cc29452d8cef0645665b4a8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 08:56:12.277074 2677 log.cc:1196] T 45cd23ad1fc74ba9b223c306d48f5205 P f2fad2d8cc29452d8cef0645665b4a8c: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-1/wal/wals/45cd23ad1fc74ba9b223c306d48f5205
I20260619 08:56:12.277288 2676 ts_tablet_manager.cc:1939] T 45cd23ad1fc74ba9b223c306d48f5205 P 1f909d146c124ed7849be361ca427112: Deleting consensus metadata
I20260619 08:56:12.277325 2677 ts_tablet_manager.cc:1939] T 45cd23ad1fc74ba9b223c306d48f5205 P f2fad2d8cc29452d8cef0645665b4a8c: Deleting consensus metadata
I20260619 08:56:12.277767 2107 catalog_manager.cc:4841] TS 1f909d146c124ed7849be361ca427112 (127.0.98.3:45917): tablet 45cd23ad1fc74ba9b223c306d48f5205 (table test-table [id=f025200df1594c309dcc4e5e33fb9013]) successfully deleted
I20260619 08:56:12.277841 2107 catalog_manager.cc:4841] TS f2fad2d8cc29452d8cef0645665b4a8c (127.0.98.2:34599): tablet 45cd23ad1fc74ba9b223c306d48f5205 (table test-table [id=f025200df1594c309dcc4e5e33fb9013]) successfully deleted
I20260619 08:56:12.278185 2628 tablet_bootstrap.cc:492] T 0859ba6f857d4263a1bd5baf244d83da P f2fad2d8cc29452d8cef0645665b4a8c: No bootstrap required, opened a new log
I20260619 08:56:12.278235 2628 ts_tablet_manager.cc:1397] T 0859ba6f857d4263a1bd5baf244d83da P f2fad2d8cc29452d8cef0645665b4a8c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260619 08:56:12.278378 2628 raft_consensus.cc:348] T 0859ba6f857d4263a1bd5baf244d83da P f2fad2d8cc29452d8cef0645665b4a8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:12.278451 2628 raft_consensus.cc:374] T 0859ba6f857d4263a1bd5baf244d83da P f2fad2d8cc29452d8cef0645665b4a8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:12.278470 2628 raft_consensus.cc:729] T 0859ba6f857d4263a1bd5baf244d83da P f2fad2d8cc29452d8cef0645665b4a8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fad2d8cc29452d8cef0645665b4a8c, State: Initialized, Role: FOLLOWER
I20260619 08:56:12.278505 2628 consensus_queue.cc:260] T 0859ba6f857d4263a1bd5baf244d83da P f2fad2d8cc29452d8cef0645665b4a8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:12.278582 2628 ts_tablet_manager.cc:1428] T 0859ba6f857d4263a1bd5baf244d83da P f2fad2d8cc29452d8cef0645665b4a8c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:12.458072 2660 raft_consensus.cc:482] T 0859ba6f857d4263a1bd5baf244d83da P facf427fe9e541b781955151f2be5909 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:56:12.458189 2660 raft_consensus.cc:504] T 0859ba6f857d4263a1bd5baf244d83da P facf427fe9e541b781955151f2be5909 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:12.458344 2660 leader_election.cc:288] T 0859ba6f857d4263a1bd5baf244d83da P facf427fe9e541b781955151f2be5909 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2fad2d8cc29452d8cef0645665b4a8c (127.0.98.2:34599), 1f909d146c124ed7849be361ca427112 (127.0.98.3:45917)
I20260619 08:56:12.458513 2633 raft_consensus.cc:482] T 0859ba6f857d4263a1bd5baf244d83da P f2fad2d8cc29452d8cef0645665b4a8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:56:12.458545 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0859ba6f857d4263a1bd5baf244d83da" candidate_uuid: "facf427fe9e541b781955151f2be5909" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" is_pre_election: true
I20260619 08:56:12.458580 2633 raft_consensus.cc:504] T 0859ba6f857d4263a1bd5baf244d83da P f2fad2d8cc29452d8cef0645665b4a8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:12.458568 2557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0859ba6f857d4263a1bd5baf244d83da" candidate_uuid: "facf427fe9e541b781955151f2be5909" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f909d146c124ed7849be361ca427112" is_pre_election: true
I20260619 08:56:12.458662 2417 raft_consensus.cc:2454] T 0859ba6f857d4263a1bd5baf244d83da P f2fad2d8cc29452d8cef0645665b4a8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate facf427fe9e541b781955151f2be5909 in term 0.
I20260619 08:56:12.458673 2557 raft_consensus.cc:2454] T 0859ba6f857d4263a1bd5baf244d83da P 1f909d146c124ed7849be361ca427112 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate facf427fe9e541b781955151f2be5909 in term 0.
I20260619 08:56:12.458694 2633 leader_election.cc:288] T 0859ba6f857d4263a1bd5baf244d83da P f2fad2d8cc29452d8cef0645665b4a8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers facf427fe9e541b781955151f2be5909 (127.0.98.1:34365), 1f909d146c124ed7849be361ca427112 (127.0.98.3:45917)
I20260619 08:56:12.458832 2211 leader_election.cc:302] T 0859ba6f857d4263a1bd5baf244d83da P facf427fe9e541b781955151f2be5909 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f909d146c124ed7849be361ca427112, facf427fe9e541b781955151f2be5909; no voters:
I20260619 08:56:12.458890 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0859ba6f857d4263a1bd5baf244d83da" candidate_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "facf427fe9e541b781955151f2be5909" is_pre_election: true
I20260619 08:56:12.458949 2660 raft_consensus.cc:2781] T 0859ba6f857d4263a1bd5baf244d83da P facf427fe9e541b781955151f2be5909 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:56:12.458999 2660 raft_consensus.cc:482] T 0859ba6f857d4263a1bd5baf244d83da P facf427fe9e541b781955151f2be5909 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:56:12.459012 2660 raft_consensus.cc:3037] T 0859ba6f857d4263a1bd5baf244d83da P facf427fe9e541b781955151f2be5909 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:12.459008 2557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0859ba6f857d4263a1bd5baf244d83da" candidate_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f909d146c124ed7849be361ca427112" is_pre_election: true
I20260619 08:56:12.459064 2557 raft_consensus.cc:2454] T 0859ba6f857d4263a1bd5baf244d83da P 1f909d146c124ed7849be361ca427112 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2fad2d8cc29452d8cef0645665b4a8c in term 0.
I20260619 08:56:12.459218 2351 leader_election.cc:302] T 0859ba6f857d4263a1bd5baf244d83da P f2fad2d8cc29452d8cef0645665b4a8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f909d146c124ed7849be361ca427112, f2fad2d8cc29452d8cef0645665b4a8c; no voters:
I20260619 08:56:12.459306 2633 raft_consensus.cc:2781] T 0859ba6f857d4263a1bd5baf244d83da P f2fad2d8cc29452d8cef0645665b4a8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:56:12.459337 2633 raft_consensus.cc:482] T 0859ba6f857d4263a1bd5baf244d83da P f2fad2d8cc29452d8cef0645665b4a8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:56:12.459352 2633 raft_consensus.cc:3037] T 0859ba6f857d4263a1bd5baf244d83da P f2fad2d8cc29452d8cef0645665b4a8c [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:12.459774 2660 raft_consensus.cc:504] T 0859ba6f857d4263a1bd5baf244d83da P facf427fe9e541b781955151f2be5909 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:12.459882 2660 leader_election.cc:288] T 0859ba6f857d4263a1bd5baf244d83da P facf427fe9e541b781955151f2be5909 [CANDIDATE]: Term 1 election: Requested vote from peers f2fad2d8cc29452d8cef0645665b4a8c (127.0.98.2:34599), 1f909d146c124ed7849be361ca427112 (127.0.98.3:45917)
I20260619 08:56:12.459897 2277 raft_consensus.cc:2379] T 0859ba6f857d4263a1bd5baf244d83da P facf427fe9e541b781955151f2be5909 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f2fad2d8cc29452d8cef0645665b4a8c in current term 1: Already voted for candidate facf427fe9e541b781955151f2be5909 in this term.
I20260619 08:56:12.460052 2633 raft_consensus.cc:504] T 0859ba6f857d4263a1bd5baf244d83da P f2fad2d8cc29452d8cef0645665b4a8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:12.460126 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0859ba6f857d4263a1bd5baf244d83da" candidate_uuid: "facf427fe9e541b781955151f2be5909" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fad2d8cc29452d8cef0645665b4a8c"
I20260619 08:56:12.460187 2417 raft_consensus.cc:2379] T 0859ba6f857d4263a1bd5baf244d83da P f2fad2d8cc29452d8cef0645665b4a8c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate facf427fe9e541b781955151f2be5909 in current term 1: Already voted for candidate f2fad2d8cc29452d8cef0645665b4a8c in this term.
I20260619 08:56:12.460197 2633 leader_election.cc:288] T 0859ba6f857d4263a1bd5baf244d83da P f2fad2d8cc29452d8cef0645665b4a8c [CANDIDATE]: Term 1 election: Requested vote from peers facf427fe9e541b781955151f2be5909 (127.0.98.1:34365), 1f909d146c124ed7849be361ca427112 (127.0.98.3:45917)
I20260619 08:56:12.460315 2557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0859ba6f857d4263a1bd5baf244d83da" candidate_uuid: "facf427fe9e541b781955151f2be5909" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f909d146c124ed7849be361ca427112"
I20260619 08:56:12.460397 2557 raft_consensus.cc:3037] T 0859ba6f857d4263a1bd5baf244d83da P 1f909d146c124ed7849be361ca427112 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:12.460464 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0859ba6f857d4263a1bd5baf244d83da" candidate_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "facf427fe9e541b781955151f2be5909"
I20260619 08:56:12.460440 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0859ba6f857d4263a1bd5baf244d83da" candidate_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f909d146c124ed7849be361ca427112"
I20260619 08:56:12.460536 2277 raft_consensus.cc:2379] T 0859ba6f857d4263a1bd5baf244d83da P facf427fe9e541b781955151f2be5909 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f2fad2d8cc29452d8cef0645665b4a8c in current term 1: Already voted for candidate facf427fe9e541b781955151f2be5909 in this term.
I20260619 08:56:12.460680 2351 leader_election.cc:302] T 0859ba6f857d4263a1bd5baf244d83da P f2fad2d8cc29452d8cef0645665b4a8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2fad2d8cc29452d8cef0645665b4a8c; no voters: 1f909d146c124ed7849be361ca427112, facf427fe9e541b781955151f2be5909
I20260619 08:56:12.460777 2633 raft_consensus.cc:2726] T 0859ba6f857d4263a1bd5baf244d83da P f2fad2d8cc29452d8cef0645665b4a8c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260619 08:56:12.461086 2557 raft_consensus.cc:2454] T 0859ba6f857d4263a1bd5baf244d83da P 1f909d146c124ed7849be361ca427112 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate facf427fe9e541b781955151f2be5909 in term 1.
I20260619 08:56:12.461283 2211 leader_election.cc:302] T 0859ba6f857d4263a1bd5baf244d83da P facf427fe9e541b781955151f2be5909 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f909d146c124ed7849be361ca427112, facf427fe9e541b781955151f2be5909; no voters: f2fad2d8cc29452d8cef0645665b4a8c
I20260619 08:56:12.461367 2660 raft_consensus.cc:2781] T 0859ba6f857d4263a1bd5baf244d83da P facf427fe9e541b781955151f2be5909 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:12.461414 2660 raft_consensus.cc:686] T 0859ba6f857d4263a1bd5baf244d83da P facf427fe9e541b781955151f2be5909 [term 1 LEADER]: Becoming Leader. State: Replica: facf427fe9e541b781955151f2be5909, State: Running, Role: LEADER
I20260619 08:56:12.461459 2660 consensus_queue.cc:237] T 0859ba6f857d4263a1bd5baf244d83da P facf427fe9e541b781955151f2be5909 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:12.461979 2120 catalog_manager.cc:5495] T 0859ba6f857d4263a1bd5baf244d83da P facf427fe9e541b781955151f2be5909 reported cstate change: term changed from 0 to 1, leader changed from <none> to facf427fe9e541b781955151f2be5909 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "facf427fe9e541b781955151f2be5909" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } health_report { overall_health: UNKNOWN } } }
I20260619 08:56:12.505853 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:12.503625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51014 (local address 127.0.98.1:34365)
0619 08:56:12.503660 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:12.503662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:12.503698 (+ 36us) server_negotiation.cc:407] Connection header received
0619 08:56:12.503767 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:12.503767 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:12.503777 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:12.503815 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:56:12.503909 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:12.504434 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:12.505198 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:12.505447 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:12.505502 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:12.505641 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:56:12.505675 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:56:12.505715 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:12.505766 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 08:56:12.505783 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 08:56:12.509764 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:12.507696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36297 (local address 127.0.98.1:34365)
0619 08:56:12.507724 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:12.507725 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:12.507732 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:56:12.507780 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:12.507780 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:12.507789 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:12.507831 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:56:12.507914 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:12.508435 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:12.509173 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:12.509359 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:12.509423 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:12.509535 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:56:12.509564 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:56:12.509606 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:12.509662 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 08:56:12.509677 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260619 08:56:12.509764 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:12.507638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34365 (local address 127.0.98.1:36297)
0619 08:56:12.507678 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:12.507689 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:12.507729 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:12.507832 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:12.507833 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:12.507834 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 08:56:12.507874 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:12.507875 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:12.508497 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:12.508498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:12.509100 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:12.509103 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:12.509430 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:12.509431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:12.509479 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:12.509485 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 08:56:12.509612 (+ 127us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 08:56:12.509613 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 08:56:12.509659 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 08:56:12.509701 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":3}
I20260619 08:56:12.511749 2557 raft_consensus.cc:1261] T 0859ba6f857d4263a1bd5baf244d83da P 1f909d146c124ed7849be361ca427112 [term 1 FOLLOWER]: Refusing update from remote peer facf427fe9e541b781955151f2be5909: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:56:12.511749 2417 raft_consensus.cc:1261] T 0859ba6f857d4263a1bd5baf244d83da P f2fad2d8cc29452d8cef0645665b4a8c [term 1 FOLLOWER]: Refusing update from remote peer facf427fe9e541b781955151f2be5909: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:56:12.511970 2661 consensus_queue.cc:1035] T 0859ba6f857d4263a1bd5baf244d83da P facf427fe9e541b781955151f2be5909 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:56:12.512113 2632 consensus_queue.cc:1035] T 0859ba6f857d4263a1bd5baf244d83da P facf427fe9e541b781955151f2be5909 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 08:56:12.516276 2654 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 08:56:14.034011 2120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:56:14.034112 2120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:56:14.035064 2120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5efeda2ae7b6428aada15b2780867507: Sending DeleteTablet for 3 replicas of tablet 0859ba6f857d4263a1bd5baf244d83da
I20260619 08:56:14.035305 2257 tablet_service.cc:1514] Processing DeleteTablet for tablet 0859ba6f857d4263a1bd5baf244d83da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51004
I20260619 08:56:14.035339 2397 tablet_service.cc:1514] Processing DeleteTablet for tablet 0859ba6f857d4263a1bd5baf244d83da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38508
I20260619 08:56:14.035507 2537 tablet_service.cc:1514] Processing DeleteTablet for tablet 0859ba6f857d4263a1bd5baf244d83da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49136
I20260619 08:56:14.035717 2707 tablet_replica.cc:330] T 0859ba6f857d4263a1bd5baf244d83da P 1f909d146c124ed7849be361ca427112: stopping tablet replica
I20260619 08:56:14.035760 2705 tablet_replica.cc:330] T 0859ba6f857d4263a1bd5baf244d83da P facf427fe9e541b781955151f2be5909: stopping tablet replica
I20260619 08:56:14.035825 2707 raft_consensus.cc:2229] T 0859ba6f857d4263a1bd5baf244d83da P 1f909d146c124ed7849be361ca427112 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:56:14.035866 2705 raft_consensus.cc:2229] T 0859ba6f857d4263a1bd5baf244d83da P facf427fe9e541b781955151f2be5909 [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:56:14.035918 2707 raft_consensus.cc:2258] T 0859ba6f857d4263a1bd5baf244d83da P 1f909d146c124ed7849be361ca427112 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:14.035991 2705 raft_consensus.cc:2258] T 0859ba6f857d4263a1bd5baf244d83da P facf427fe9e541b781955151f2be5909 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:14.036227 2707 ts_tablet_manager.cc:1905] T 0859ba6f857d4263a1bd5baf244d83da P 1f909d146c124ed7849be361ca427112: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:14.036283 2705 ts_tablet_manager.cc:1905] T 0859ba6f857d4263a1bd5baf244d83da P facf427fe9e541b781955151f2be5909: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:14.037952 2706 tablet_replica.cc:330] T 0859ba6f857d4263a1bd5baf244d83da P f2fad2d8cc29452d8cef0645665b4a8c: stopping tablet replica
I20260619 08:56:14.037971 2705 ts_tablet_manager.cc:1918] T 0859ba6f857d4263a1bd5baf244d83da P facf427fe9e541b781955151f2be5909: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 08:56:14.037987 2707 ts_tablet_manager.cc:1918] T 0859ba6f857d4263a1bd5baf244d83da P 1f909d146c124ed7849be361ca427112: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 08:56:14.038013 2705 log.cc:1196] T 0859ba6f857d4263a1bd5baf244d83da P facf427fe9e541b781955151f2be5909: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-0/wal/wals/0859ba6f857d4263a1bd5baf244d83da
I20260619 08:56:14.038031 2707 log.cc:1196] T 0859ba6f857d4263a1bd5baf244d83da P 1f909d146c124ed7849be361ca427112: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-2/wal/wals/0859ba6f857d4263a1bd5baf244d83da
I20260619 08:56:14.038065 2706 raft_consensus.cc:2229] T 0859ba6f857d4263a1bd5baf244d83da P f2fad2d8cc29452d8cef0645665b4a8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:56:14.038137 2706 raft_consensus.cc:2258] T 0859ba6f857d4263a1bd5baf244d83da P f2fad2d8cc29452d8cef0645665b4a8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:14.038264 2707 ts_tablet_manager.cc:1939] T 0859ba6f857d4263a1bd5baf244d83da P 1f909d146c124ed7849be361ca427112: Deleting consensus metadata
I20260619 08:56:14.038374 2705 ts_tablet_manager.cc:1939] T 0859ba6f857d4263a1bd5baf244d83da P facf427fe9e541b781955151f2be5909: Deleting consensus metadata
I20260619 08:56:14.038414 2706 ts_tablet_manager.cc:1905] T 0859ba6f857d4263a1bd5baf244d83da P f2fad2d8cc29452d8cef0645665b4a8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:14.038544 2107 catalog_manager.cc:4841] TS 1f909d146c124ed7849be361ca427112 (127.0.98.3:45917): tablet 0859ba6f857d4263a1bd5baf244d83da (table test-table [id=898cb918835d404eb62cc61d258cf604]) successfully deleted
I20260619 08:56:14.038631 2107 catalog_manager.cc:4841] TS facf427fe9e541b781955151f2be5909 (127.0.98.1:34365): tablet 0859ba6f857d4263a1bd5baf244d83da (table test-table [id=898cb918835d404eb62cc61d258cf604]) successfully deleted
I20260619 08:56:14.039659 2706 ts_tablet_manager.cc:1918] T 0859ba6f857d4263a1bd5baf244d83da P f2fad2d8cc29452d8cef0645665b4a8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 08:56:14.039716 2706 log.cc:1196] T 0859ba6f857d4263a1bd5baf244d83da P f2fad2d8cc29452d8cef0645665b4a8c: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-1/wal/wals/0859ba6f857d4263a1bd5baf244d83da
I20260619 08:56:14.039888 2706 ts_tablet_manager.cc:1939] T 0859ba6f857d4263a1bd5baf244d83da P f2fad2d8cc29452d8cef0645665b4a8c: Deleting consensus metadata
I20260619 08:56:14.040319 2107 catalog_manager.cc:4841] TS f2fad2d8cc29452d8cef0645665b4a8c (127.0.98.2:34599): tablet 0859ba6f857d4263a1bd5baf244d83da (table test-table [id=898cb918835d404eb62cc61d258cf604]) successfully deleted
Jun 19 08:56:14 dist-test-slave-h4g5 krb5kdc[2075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859374, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 08:56:14 dist-test-slave-h4g5 krb5kdc[2075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859374, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:14.059459 2715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:14.052482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39210 (local address 127.0.98.62:43031)
0619 08:56:14.052612 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:14.052614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:14.052622 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:56:14.052650 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:14.052651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:14.052661 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:14.052722 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:14.053362 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:14.053879 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:14.054645 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:14.054900 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:14.054966 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:14.057217 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:14.057218 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:14.057219 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:14.057225 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:14.058567 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:14.058872 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:14.058872 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:14.058873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:14.058922 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:14.059053 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:14.059054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:14.059054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:14.059151 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:14.059219 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:14.059323 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 08:56:14.059385 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":71,"threads_started":1}
I20260619 08:56:14.061021 2120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39210:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:56:14.061192 2120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:56:14.062407 2257 tablet_service.cc:1467] Processing CreateTablet for tablet b1df067f346a47f492212704602e2270 (DEFAULT_TABLE table=test-table [id=dd19d0b8297447d7ac959851ceadcb7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:14.062425 2537 tablet_service.cc:1467] Processing CreateTablet for tablet b1df067f346a47f492212704602e2270 (DEFAULT_TABLE table=test-table [id=dd19d0b8297447d7ac959851ceadcb7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:14.062425 2397 tablet_service.cc:1467] Processing CreateTablet for tablet b1df067f346a47f492212704602e2270 (DEFAULT_TABLE table=test-table [id=dd19d0b8297447d7ac959851ceadcb7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:14.062544 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1df067f346a47f492212704602e2270. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:14.062551 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1df067f346a47f492212704602e2270. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:14.062547 2537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1df067f346a47f492212704602e2270. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:14.063755 2716 tablet_bootstrap.cc:492] T b1df067f346a47f492212704602e2270 P facf427fe9e541b781955151f2be5909: Bootstrap starting.
I20260619 08:56:14.063931 2718 tablet_bootstrap.cc:492] T b1df067f346a47f492212704602e2270 P f2fad2d8cc29452d8cef0645665b4a8c: Bootstrap starting.
I20260619 08:56:14.064148 2717 tablet_bootstrap.cc:492] T b1df067f346a47f492212704602e2270 P 1f909d146c124ed7849be361ca427112: Bootstrap starting.
I20260619 08:56:14.064204 2716 tablet_bootstrap.cc:654] T b1df067f346a47f492212704602e2270 P facf427fe9e541b781955151f2be5909: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:14.064307 2718 tablet_bootstrap.cc:654] T b1df067f346a47f492212704602e2270 P f2fad2d8cc29452d8cef0645665b4a8c: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:14.064538 2717 tablet_bootstrap.cc:654] T b1df067f346a47f492212704602e2270 P 1f909d146c124ed7849be361ca427112: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:14.064867 2716 tablet_bootstrap.cc:492] T b1df067f346a47f492212704602e2270 P facf427fe9e541b781955151f2be5909: No bootstrap required, opened a new log
I20260619 08:56:14.064911 2716 ts_tablet_manager.cc:1397] T b1df067f346a47f492212704602e2270 P facf427fe9e541b781955151f2be5909: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 08:56:14.065002 2718 tablet_bootstrap.cc:492] T b1df067f346a47f492212704602e2270 P f2fad2d8cc29452d8cef0645665b4a8c: No bootstrap required, opened a new log
I20260619 08:56:14.065054 2718 ts_tablet_manager.cc:1397] T b1df067f346a47f492212704602e2270 P f2fad2d8cc29452d8cef0645665b4a8c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:14.065022 2716 raft_consensus.cc:348] T b1df067f346a47f492212704602e2270 P facf427fe9e541b781955151f2be5909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:14.065088 2716 raft_consensus.cc:374] T b1df067f346a47f492212704602e2270 P facf427fe9e541b781955151f2be5909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:14.065115 2716 raft_consensus.cc:729] T b1df067f346a47f492212704602e2270 P facf427fe9e541b781955151f2be5909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: facf427fe9e541b781955151f2be5909, State: Initialized, Role: FOLLOWER
I20260619 08:56:14.065162 2716 consensus_queue.cc:260] T b1df067f346a47f492212704602e2270 P facf427fe9e541b781955151f2be5909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:14.065163 2718 raft_consensus.cc:348] T b1df067f346a47f492212704602e2270 P f2fad2d8cc29452d8cef0645665b4a8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:14.065234 2718 raft_consensus.cc:374] T b1df067f346a47f492212704602e2270 P f2fad2d8cc29452d8cef0645665b4a8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:14.065259 2718 raft_consensus.cc:729] T b1df067f346a47f492212704602e2270 P f2fad2d8cc29452d8cef0645665b4a8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fad2d8cc29452d8cef0645665b4a8c, State: Initialized, Role: FOLLOWER
I20260619 08:56:14.065256 2716 ts_tablet_manager.cc:1428] T b1df067f346a47f492212704602e2270 P facf427fe9e541b781955151f2be5909: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:14.065310 2718 consensus_queue.cc:260] T b1df067f346a47f492212704602e2270 P f2fad2d8cc29452d8cef0645665b4a8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:14.065465 2718 ts_tablet_manager.cc:1428] T b1df067f346a47f492212704602e2270 P f2fad2d8cc29452d8cef0645665b4a8c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:14.065828 2717 tablet_bootstrap.cc:492] T b1df067f346a47f492212704602e2270 P 1f909d146c124ed7849be361ca427112: No bootstrap required, opened a new log
I20260619 08:56:14.065876 2717 ts_tablet_manager.cc:1397] T b1df067f346a47f492212704602e2270 P 1f909d146c124ed7849be361ca427112: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:56:14.066007 2717 raft_consensus.cc:348] T b1df067f346a47f492212704602e2270 P 1f909d146c124ed7849be361ca427112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:14.066076 2717 raft_consensus.cc:374] T b1df067f346a47f492212704602e2270 P 1f909d146c124ed7849be361ca427112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:14.066097 2717 raft_consensus.cc:729] T b1df067f346a47f492212704602e2270 P 1f909d146c124ed7849be361ca427112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f909d146c124ed7849be361ca427112, State: Initialized, Role: FOLLOWER
I20260619 08:56:14.066152 2717 consensus_queue.cc:260] T b1df067f346a47f492212704602e2270 P 1f909d146c124ed7849be361ca427112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:14.066315 2717 ts_tablet_manager.cc:1428] T b1df067f346a47f492212704602e2270 P 1f909d146c124ed7849be361ca427112: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:14.150192 2722 raft_consensus.cc:482] T b1df067f346a47f492212704602e2270 P f2fad2d8cc29452d8cef0645665b4a8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:56:14.150302 2722 raft_consensus.cc:504] T b1df067f346a47f492212704602e2270 P f2fad2d8cc29452d8cef0645665b4a8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:14.150435 2722 leader_election.cc:288] T b1df067f346a47f492212704602e2270 P f2fad2d8cc29452d8cef0645665b4a8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers facf427fe9e541b781955151f2be5909 (127.0.98.1:34365), 1f909d146c124ed7849be361ca427112 (127.0.98.3:45917)
I20260619 08:56:14.150709 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1df067f346a47f492212704602e2270" candidate_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "facf427fe9e541b781955151f2be5909" is_pre_election: true
I20260619 08:56:14.150806 2277 raft_consensus.cc:2454] T b1df067f346a47f492212704602e2270 P facf427fe9e541b781955151f2be5909 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2fad2d8cc29452d8cef0645665b4a8c in term 0.
I20260619 08:56:14.150733 2557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1df067f346a47f492212704602e2270" candidate_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f909d146c124ed7849be361ca427112" is_pre_election: true
I20260619 08:56:14.150888 2557 raft_consensus.cc:2454] T b1df067f346a47f492212704602e2270 P 1f909d146c124ed7849be361ca427112 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2fad2d8cc29452d8cef0645665b4a8c in term 0.
I20260619 08:56:14.151008 2351 leader_election.cc:302] T b1df067f346a47f492212704602e2270 P f2fad2d8cc29452d8cef0645665b4a8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f2fad2d8cc29452d8cef0645665b4a8c, facf427fe9e541b781955151f2be5909; no voters:
I20260619 08:56:14.151146 2722 raft_consensus.cc:2781] T b1df067f346a47f492212704602e2270 P f2fad2d8cc29452d8cef0645665b4a8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:56:14.151207 2722 raft_consensus.cc:482] T b1df067f346a47f492212704602e2270 P f2fad2d8cc29452d8cef0645665b4a8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:56:14.151223 2722 raft_consensus.cc:3037] T b1df067f346a47f492212704602e2270 P f2fad2d8cc29452d8cef0645665b4a8c [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:14.151960 2722 raft_consensus.cc:504] T b1df067f346a47f492212704602e2270 P f2fad2d8cc29452d8cef0645665b4a8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:14.152127 2722 leader_election.cc:288] T b1df067f346a47f492212704602e2270 P f2fad2d8cc29452d8cef0645665b4a8c [CANDIDATE]: Term 1 election: Requested vote from peers facf427fe9e541b781955151f2be5909 (127.0.98.1:34365), 1f909d146c124ed7849be361ca427112 (127.0.98.3:45917)
I20260619 08:56:14.152280 2557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1df067f346a47f492212704602e2270" candidate_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f909d146c124ed7849be361ca427112"
I20260619 08:56:14.152287 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1df067f346a47f492212704602e2270" candidate_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "facf427fe9e541b781955151f2be5909"
I20260619 08:56:14.152362 2277 raft_consensus.cc:3037] T b1df067f346a47f492212704602e2270 P facf427fe9e541b781955151f2be5909 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:14.152361 2557 raft_consensus.cc:3037] T b1df067f346a47f492212704602e2270 P 1f909d146c124ed7849be361ca427112 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:14.153105 2557 raft_consensus.cc:2454] T b1df067f346a47f492212704602e2270 P 1f909d146c124ed7849be361ca427112 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2fad2d8cc29452d8cef0645665b4a8c in term 1.
I20260619 08:56:14.153105 2277 raft_consensus.cc:2454] T b1df067f346a47f492212704602e2270 P facf427fe9e541b781955151f2be5909 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2fad2d8cc29452d8cef0645665b4a8c in term 1.
I20260619 08:56:14.153321 2351 leader_election.cc:302] T b1df067f346a47f492212704602e2270 P f2fad2d8cc29452d8cef0645665b4a8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f909d146c124ed7849be361ca427112, f2fad2d8cc29452d8cef0645665b4a8c; no voters:
I20260619 08:56:14.153429 2722 raft_consensus.cc:2781] T b1df067f346a47f492212704602e2270 P f2fad2d8cc29452d8cef0645665b4a8c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:14.153569 2722 raft_consensus.cc:686] T b1df067f346a47f492212704602e2270 P f2fad2d8cc29452d8cef0645665b4a8c [term 1 LEADER]: Becoming Leader. State: Replica: f2fad2d8cc29452d8cef0645665b4a8c, State: Running, Role: LEADER
I20260619 08:56:14.153613 2722 consensus_queue.cc:237] T b1df067f346a47f492212704602e2270 P f2fad2d8cc29452d8cef0645665b4a8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:14.154238 2119 catalog_manager.cc:5495] T b1df067f346a47f492212704602e2270 P f2fad2d8cc29452d8cef0645665b4a8c reported cstate change: term changed from 0 to 1, leader changed from <none> to f2fad2d8cc29452d8cef0645665b4a8c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } health_report { overall_health: UNKNOWN } } }
I20260619 08:56:14.184051 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:14.181624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38528 (local address 127.0.98.2:34599)
0619 08:56:14.181781 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:14.181782 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:14.181788 (+ 6us) server_negotiation.cc:407] Connection header received
0619 08:56:14.181807 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:14.181808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:14.181819 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:14.181881 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:56:14.182025 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:14.182580 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:14.183314 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:14.183593 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:14.183651 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:14.183797 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:56:14.183835 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:56:14.183885 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:14.183938 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 08:56:14.183962 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":103,"threads_started":1}
I20260619 08:56:14.186767 2557 raft_consensus.cc:1261] T b1df067f346a47f492212704602e2270 P 1f909d146c124ed7849be361ca427112 [term 1 FOLLOWER]: Refusing update from remote peer f2fad2d8cc29452d8cef0645665b4a8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:56:14.186810 2277 raft_consensus.cc:1261] T b1df067f346a47f492212704602e2270 P facf427fe9e541b781955151f2be5909 [term 1 FOLLOWER]: Refusing update from remote peer f2fad2d8cc29452d8cef0645665b4a8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:56:14.186993 2724 consensus_queue.cc:1035] T b1df067f346a47f492212704602e2270 P f2fad2d8cc29452d8cef0645665b4a8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:56:14.187114 2722 consensus_queue.cc:1035] T b1df067f346a47f492212704602e2270 P f2fad2d8cc29452d8cef0645665b4a8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:56:14.196812 2119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:56:14.196910 2119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:56:14.197480 2119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5efeda2ae7b6428aada15b2780867507: Sending DeleteTablet for 3 replicas of tablet b1df067f346a47f492212704602e2270
I20260619 08:56:14.197741 2397 tablet_service.cc:1514] Processing DeleteTablet for tablet b1df067f346a47f492212704602e2270 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38508
I20260619 08:56:14.197741 2537 tablet_service.cc:1514] Processing DeleteTablet for tablet b1df067f346a47f492212704602e2270 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49136
I20260619 08:56:14.197746 2257 tablet_service.cc:1514] Processing DeleteTablet for tablet b1df067f346a47f492212704602e2270 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51004
I20260619 08:56:14.197835 2705 tablet_replica.cc:330] T b1df067f346a47f492212704602e2270 P facf427fe9e541b781955151f2be5909: stopping tablet replica
I20260619 08:56:14.197835 2706 tablet_replica.cc:330] T b1df067f346a47f492212704602e2270 P f2fad2d8cc29452d8cef0645665b4a8c: stopping tablet replica
I20260619 08:56:14.197852 2707 tablet_replica.cc:330] T b1df067f346a47f492212704602e2270 P 1f909d146c124ed7849be361ca427112: stopping tablet replica
I20260619 08:56:14.197881 2705 raft_consensus.cc:2229] T b1df067f346a47f492212704602e2270 P facf427fe9e541b781955151f2be5909 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:56:14.197886 2706 raft_consensus.cc:2229] T b1df067f346a47f492212704602e2270 P f2fad2d8cc29452d8cef0645665b4a8c [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:56:14.197899 2707 raft_consensus.cc:2229] T b1df067f346a47f492212704602e2270 P 1f909d146c124ed7849be361ca427112 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:56:14.197934 2707 raft_consensus.cc:2258] T b1df067f346a47f492212704602e2270 P 1f909d146c124ed7849be361ca427112 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:14.197911 2705 raft_consensus.cc:2258] T b1df067f346a47f492212704602e2270 P facf427fe9e541b781955151f2be5909 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:14.197939 2706 raft_consensus.cc:2258] T b1df067f346a47f492212704602e2270 P f2fad2d8cc29452d8cef0645665b4a8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:14.198253 2707 ts_tablet_manager.cc:1905] T b1df067f346a47f492212704602e2270 P 1f909d146c124ed7849be361ca427112: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:14.198254 2706 ts_tablet_manager.cc:1905] T b1df067f346a47f492212704602e2270 P f2fad2d8cc29452d8cef0645665b4a8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:14.198297 2705 ts_tablet_manager.cc:1905] T b1df067f346a47f492212704602e2270 P facf427fe9e541b781955151f2be5909: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:14.199705 2707 ts_tablet_manager.cc:1918] T b1df067f346a47f492212704602e2270 P 1f909d146c124ed7849be361ca427112: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 08:56:14.199718 2705 ts_tablet_manager.cc:1918] T b1df067f346a47f492212704602e2270 P facf427fe9e541b781955151f2be5909: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 08:56:14.199739 2706 ts_tablet_manager.cc:1918] T b1df067f346a47f492212704602e2270 P f2fad2d8cc29452d8cef0645665b4a8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 08:56:14.199762 2707 log.cc:1196] T b1df067f346a47f492212704602e2270 P 1f909d146c124ed7849be361ca427112: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-2/wal/wals/b1df067f346a47f492212704602e2270
I20260619 08:56:14.199776 2705 log.cc:1196] T b1df067f346a47f492212704602e2270 P facf427fe9e541b781955151f2be5909: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-0/wal/wals/b1df067f346a47f492212704602e2270
I20260619 08:56:14.199779 2706 log.cc:1196] T b1df067f346a47f492212704602e2270 P f2fad2d8cc29452d8cef0645665b4a8c: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-1/wal/wals/b1df067f346a47f492212704602e2270
I20260619 08:56:14.199702 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39210:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:56:14.199846 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:56:14.199956 2707 ts_tablet_manager.cc:1939] T b1df067f346a47f492212704602e2270 P 1f909d146c124ed7849be361ca427112: Deleting consensus metadata
I20260619 08:56:14.199976 2705 ts_tablet_manager.cc:1939] T b1df067f346a47f492212704602e2270 P facf427fe9e541b781955151f2be5909: Deleting consensus metadata
I20260619 08:56:14.199983 2706 ts_tablet_manager.cc:1939] T b1df067f346a47f492212704602e2270 P f2fad2d8cc29452d8cef0645665b4a8c: Deleting consensus metadata
I20260619 08:56:14.200361 2107 catalog_manager.cc:4841] TS facf427fe9e541b781955151f2be5909 (127.0.98.1:34365): tablet b1df067f346a47f492212704602e2270 (table test-table [id=dd19d0b8297447d7ac959851ceadcb7d]) successfully deleted
I20260619 08:56:14.200441 2107 catalog_manager.cc:4841] TS 1f909d146c124ed7849be361ca427112 (127.0.98.3:45917): tablet b1df067f346a47f492212704602e2270 (table test-table [id=dd19d0b8297447d7ac959851ceadcb7d]) successfully deleted
I20260619 08:56:14.200687 2107 catalog_manager.cc:4841] TS f2fad2d8cc29452d8cef0645665b4a8c (127.0.98.2:34599): tablet b1df067f346a47f492212704602e2270 (table test-table [id=dd19d0b8297447d7ac959851ceadcb7d]) successfully deleted
I20260619 08:56:14.201269 2257 tablet_service.cc:1467] Processing CreateTablet for tablet 05553009df464eb1967ad8ef3c48264e (DEFAULT_TABLE table=test-table [id=cd4d92a0b511483eb385742f1bf8fe60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:14.201287 2397 tablet_service.cc:1467] Processing CreateTablet for tablet 05553009df464eb1967ad8ef3c48264e (DEFAULT_TABLE table=test-table [id=cd4d92a0b511483eb385742f1bf8fe60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:14.201325 2537 tablet_service.cc:1467] Processing CreateTablet for tablet 05553009df464eb1967ad8ef3c48264e (DEFAULT_TABLE table=test-table [id=cd4d92a0b511483eb385742f1bf8fe60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:14.201380 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05553009df464eb1967ad8ef3c48264e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:14.201408 2537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05553009df464eb1967ad8ef3c48264e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:14.201392 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05553009df464eb1967ad8ef3c48264e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:14.202281 2718 tablet_bootstrap.cc:492] T 05553009df464eb1967ad8ef3c48264e P f2fad2d8cc29452d8cef0645665b4a8c: Bootstrap starting.
I20260619 08:56:14.202507 2717 tablet_bootstrap.cc:492] T 05553009df464eb1967ad8ef3c48264e P 1f909d146c124ed7849be361ca427112: Bootstrap starting.
I20260619 08:56:14.202581 2718 tablet_bootstrap.cc:654] T 05553009df464eb1967ad8ef3c48264e P f2fad2d8cc29452d8cef0645665b4a8c: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:14.202836 2716 tablet_bootstrap.cc:492] T 05553009df464eb1967ad8ef3c48264e P facf427fe9e541b781955151f2be5909: Bootstrap starting.
I20260619 08:56:14.202872 2717 tablet_bootstrap.cc:654] T 05553009df464eb1967ad8ef3c48264e P 1f909d146c124ed7849be361ca427112: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:14.203123 2718 tablet_bootstrap.cc:492] T 05553009df464eb1967ad8ef3c48264e P f2fad2d8cc29452d8cef0645665b4a8c: No bootstrap required, opened a new log
I20260619 08:56:14.203142 2716 tablet_bootstrap.cc:654] T 05553009df464eb1967ad8ef3c48264e P facf427fe9e541b781955151f2be5909: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:14.203166 2718 ts_tablet_manager.cc:1397] T 05553009df464eb1967ad8ef3c48264e P f2fad2d8cc29452d8cef0645665b4a8c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:14.203269 2718 raft_consensus.cc:348] T 05553009df464eb1967ad8ef3c48264e P f2fad2d8cc29452d8cef0645665b4a8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:14.203338 2718 raft_consensus.cc:374] T 05553009df464eb1967ad8ef3c48264e P f2fad2d8cc29452d8cef0645665b4a8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:14.203356 2718 raft_consensus.cc:729] T 05553009df464eb1967ad8ef3c48264e P f2fad2d8cc29452d8cef0645665b4a8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fad2d8cc29452d8cef0645665b4a8c, State: Initialized, Role: FOLLOWER
I20260619 08:56:14.203397 2718 consensus_queue.cc:260] T 05553009df464eb1967ad8ef3c48264e P f2fad2d8cc29452d8cef0645665b4a8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:14.203511 2718 ts_tablet_manager.cc:1428] T 05553009df464eb1967ad8ef3c48264e P f2fad2d8cc29452d8cef0645665b4a8c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:14.203619 2717 tablet_bootstrap.cc:492] T 05553009df464eb1967ad8ef3c48264e P 1f909d146c124ed7849be361ca427112: No bootstrap required, opened a new log
I20260619 08:56:14.203660 2717 ts_tablet_manager.cc:1397] T 05553009df464eb1967ad8ef3c48264e P 1f909d146c124ed7849be361ca427112: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:14.203682 2716 tablet_bootstrap.cc:492] T 05553009df464eb1967ad8ef3c48264e P facf427fe9e541b781955151f2be5909: No bootstrap required, opened a new log
I20260619 08:56:14.203719 2716 ts_tablet_manager.cc:1397] T 05553009df464eb1967ad8ef3c48264e P facf427fe9e541b781955151f2be5909: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 08:56:14.203791 2717 raft_consensus.cc:348] T 05553009df464eb1967ad8ef3c48264e P 1f909d146c124ed7849be361ca427112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:14.203850 2717 raft_consensus.cc:374] T 05553009df464eb1967ad8ef3c48264e P 1f909d146c124ed7849be361ca427112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:14.203867 2717 raft_consensus.cc:729] T 05553009df464eb1967ad8ef3c48264e P 1f909d146c124ed7849be361ca427112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f909d146c124ed7849be361ca427112, State: Initialized, Role: FOLLOWER
I20260619 08:56:14.203869 2716 raft_consensus.cc:348] T 05553009df464eb1967ad8ef3c48264e P facf427fe9e541b781955151f2be5909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:14.203910 2717 consensus_queue.cc:260] T 05553009df464eb1967ad8ef3c48264e P 1f909d146c124ed7849be361ca427112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:14.203936 2716 raft_consensus.cc:374] T 05553009df464eb1967ad8ef3c48264e P facf427fe9e541b781955151f2be5909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:14.203955 2716 raft_consensus.cc:729] T 05553009df464eb1967ad8ef3c48264e P facf427fe9e541b781955151f2be5909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: facf427fe9e541b781955151f2be5909, State: Initialized, Role: FOLLOWER
I20260619 08:56:14.203994 2717 ts_tablet_manager.cc:1428] T 05553009df464eb1967ad8ef3c48264e P 1f909d146c124ed7849be361ca427112: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:14.203989 2716 consensus_queue.cc:260] T 05553009df464eb1967ad8ef3c48264e P facf427fe9e541b781955151f2be5909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:14.204069 2716 ts_tablet_manager.cc:1428] T 05553009df464eb1967ad8ef3c48264e P facf427fe9e541b781955151f2be5909: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:14.253216 2723 raft_consensus.cc:482] T 05553009df464eb1967ad8ef3c48264e P 1f909d146c124ed7849be361ca427112 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:56:14.253311 2723 raft_consensus.cc:504] T 05553009df464eb1967ad8ef3c48264e P 1f909d146c124ed7849be361ca427112 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:14.253582 2723 leader_election.cc:288] T 05553009df464eb1967ad8ef3c48264e P 1f909d146c124ed7849be361ca427112 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers facf427fe9e541b781955151f2be5909 (127.0.98.1:34365), f2fad2d8cc29452d8cef0645665b4a8c (127.0.98.2:34599)
I20260619 08:56:14.256505 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:14.254073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55259 (local address 127.0.98.2:34599)
0619 08:56:14.254115 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:14.254117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:14.254200 (+ 83us) server_negotiation.cc:407] Connection header received
0619 08:56:14.254254 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:14.254255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:14.254264 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:14.254309 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:14.254452 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:14.254932 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:14.255823 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:14.256201 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:14.256260 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:14.256271 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:14.256398 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 08:56:14.256432 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 08:56:14.256553 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:14.254012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34599 (local address 127.0.98.3:55259)
0619 08:56:14.254142 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:14.254149 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:14.254188 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:14.254321 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:14.254322 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:14.254323 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:14.254392 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:14.254393 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:14.255028 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:14.255029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:14.255737 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:14.255738 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:14.256271 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:14.256271 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:14.256336 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:14.256339 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:56:14.256383 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 08:56:14.256425 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":77,"threads_started":1}
I20260619 08:56:14.256582 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:14.253793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34365 (local address 127.0.98.3:43653)
0619 08:56:14.253915 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:14.253926 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:14.254027 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:14.254197 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:14.254198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:14.254201 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:14.254275 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:14.254277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:14.254917 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:14.254918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:14.255735 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:14.255737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:14.256317 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:14.256317 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:14.256381 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:14.256384 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:56:14.256428 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 08:56:14.256469 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260619 08:56:14.256567 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:14.253892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43653 (local address 127.0.98.1:34365)
0619 08:56:14.254003 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:14.254005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:14.254015 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:56:14.254095 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:14.254095 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:14.254106 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:14.254182 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:14.254356 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:14.254841 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:14.255826 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:14.256248 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:14.256309 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:14.256320 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:14.256441 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 08:56:14.256493 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":53,"threads_started":1}
I20260619 08:56:14.256806 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05553009df464eb1967ad8ef3c48264e" candidate_uuid: "1f909d146c124ed7849be361ca427112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" is_pre_election: true
I20260619 08:56:14.256866 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05553009df464eb1967ad8ef3c48264e" candidate_uuid: "1f909d146c124ed7849be361ca427112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "facf427fe9e541b781955151f2be5909" is_pre_election: true
I20260619 08:56:14.256907 2417 raft_consensus.cc:2454] T 05553009df464eb1967ad8ef3c48264e P f2fad2d8cc29452d8cef0645665b4a8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f909d146c124ed7849be361ca427112 in term 0.
I20260619 08:56:14.256946 2277 raft_consensus.cc:2454] T 05553009df464eb1967ad8ef3c48264e P facf427fe9e541b781955151f2be5909 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f909d146c124ed7849be361ca427112 in term 0.
I20260619 08:56:14.257091 2491 leader_election.cc:302] T 05553009df464eb1967ad8ef3c48264e P 1f909d146c124ed7849be361ca427112 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f909d146c124ed7849be361ca427112, f2fad2d8cc29452d8cef0645665b4a8c; no voters:
I20260619 08:56:14.257221 2723 raft_consensus.cc:2781] T 05553009df464eb1967ad8ef3c48264e P 1f909d146c124ed7849be361ca427112 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:56:14.257259 2723 raft_consensus.cc:482] T 05553009df464eb1967ad8ef3c48264e P 1f909d146c124ed7849be361ca427112 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:56:14.257287 2723 raft_consensus.cc:3037] T 05553009df464eb1967ad8ef3c48264e P 1f909d146c124ed7849be361ca427112 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:14.257952 2723 raft_consensus.cc:504] T 05553009df464eb1967ad8ef3c48264e P 1f909d146c124ed7849be361ca427112 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:14.258059 2723 leader_election.cc:288] T 05553009df464eb1967ad8ef3c48264e P 1f909d146c124ed7849be361ca427112 [CANDIDATE]: Term 1 election: Requested vote from peers facf427fe9e541b781955151f2be5909 (127.0.98.1:34365), f2fad2d8cc29452d8cef0645665b4a8c (127.0.98.2:34599)
I20260619 08:56:14.258205 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05553009df464eb1967ad8ef3c48264e" candidate_uuid: "1f909d146c124ed7849be361ca427112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "facf427fe9e541b781955151f2be5909"
I20260619 08:56:14.258283 2277 raft_consensus.cc:3037] T 05553009df464eb1967ad8ef3c48264e P facf427fe9e541b781955151f2be5909 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:14.258252 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05553009df464eb1967ad8ef3c48264e" candidate_uuid: "1f909d146c124ed7849be361ca427112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2fad2d8cc29452d8cef0645665b4a8c"
I20260619 08:56:14.258334 2417 raft_consensus.cc:3037] T 05553009df464eb1967ad8ef3c48264e P f2fad2d8cc29452d8cef0645665b4a8c [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:14.258836 2277 raft_consensus.cc:2454] T 05553009df464eb1967ad8ef3c48264e P facf427fe9e541b781955151f2be5909 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f909d146c124ed7849be361ca427112 in term 1.
I20260619 08:56:14.258939 2417 raft_consensus.cc:2454] T 05553009df464eb1967ad8ef3c48264e P f2fad2d8cc29452d8cef0645665b4a8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f909d146c124ed7849be361ca427112 in term 1.
I20260619 08:56:14.258993 2491 leader_election.cc:302] T 05553009df464eb1967ad8ef3c48264e P 1f909d146c124ed7849be361ca427112 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f909d146c124ed7849be361ca427112, facf427fe9e541b781955151f2be5909; no voters:
I20260619 08:56:14.259078 2723 raft_consensus.cc:2781] T 05553009df464eb1967ad8ef3c48264e P 1f909d146c124ed7849be361ca427112 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:14.259215 2723 raft_consensus.cc:686] T 05553009df464eb1967ad8ef3c48264e P 1f909d146c124ed7849be361ca427112 [term 1 LEADER]: Becoming Leader. State: Replica: 1f909d146c124ed7849be361ca427112, State: Running, Role: LEADER
I20260619 08:56:14.259280 2723 consensus_queue.cc:237] T 05553009df464eb1967ad8ef3c48264e P 1f909d146c124ed7849be361ca427112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } }
I20260619 08:56:14.259935 2119 catalog_manager.cc:5495] T 05553009df464eb1967ad8ef3c48264e P 1f909d146c124ed7849be361ca427112 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f909d146c124ed7849be361ca427112 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1f909d146c124ed7849be361ca427112" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f909d146c124ed7849be361ca427112" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } health_report { overall_health: HEALTHY } } }
I20260619 08:56:14.268210 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:14.265393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49144 (local address 127.0.98.3:45917)
0619 08:56:14.265545 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:14.265546 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:14.265552 (+ 6us) server_negotiation.cc:407] Connection header received
0619 08:56:14.265582 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:14.265583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:14.265599 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:14.265671 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:56:14.265793 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:14.266297 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:14.267009 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:14.267654 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:14.267717 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:14.267866 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:56:14.267978 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:56:14.268050 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:14.268100 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 08:56:14.268121 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
I20260619 08:56:14.272850 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:14.270673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48827 (local address 127.0.98.1:34365)
0619 08:56:14.270727 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:14.270729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:14.270738 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:56:14.270790 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:14.270791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:14.270802 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:14.270845 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:56:14.270961 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:14.271410 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:14.272238 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:14.272428 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:14.272479 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:14.272627 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:56:14.272652 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:56:14.272685 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:14.272776 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 08:56:14.272794 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 08:56:14.272933 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:14.270498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34365 (local address 127.0.98.3:48827)
0619 08:56:14.270691 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:14.270702 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:14.270747 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:14.270853 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:14.270854 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:14.270855 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 08:56:14.270910 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:14.270911 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:14.271489 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:14.271490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:14.272169 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:14.272171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:14.272496 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:14.272497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:14.272555 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:14.272567 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 08:56:14.272718 (+ 151us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 08:56:14.272720 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:56:14.272773 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 08:56:14.272811 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":87,"threads_started":1}
I20260619 08:56:14.274861 2417 raft_consensus.cc:1261] T 05553009df464eb1967ad8ef3c48264e P f2fad2d8cc29452d8cef0645665b4a8c [term 1 FOLLOWER]: Refusing update from remote peer 1f909d146c124ed7849be361ca427112: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:56:14.274896 2277 raft_consensus.cc:1261] T 05553009df464eb1967ad8ef3c48264e P facf427fe9e541b781955151f2be5909 [term 1 FOLLOWER]: Refusing update from remote peer 1f909d146c124ed7849be361ca427112: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:56:14.275127 2738 consensus_queue.cc:1035] T 05553009df464eb1967ad8ef3c48264e P 1f909d146c124ed7849be361ca427112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2fad2d8cc29452d8cef0645665b4a8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:56:14.275259 2723 consensus_queue.cc:1035] T 05553009df464eb1967ad8ef3c48264e P 1f909d146c124ed7849be361ca427112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "facf427fe9e541b781955151f2be5909" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:56:14.282016 2748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:14.279574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45917 (local address 127.0.98.1:51423)
0619 08:56:14.279683 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:14.279695 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:14.279750 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:14.279893 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:14.279894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:14.279896 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 08:56:14.279947 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:14.279949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:14.280587 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:14.280588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:14.281224 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:14.281226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:14.281645 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:14.281646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:14.281704 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:14.281709 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 08:56:14.281853 (+ 144us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 08:56:14.281855 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:56:14.281901 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 08:56:14.281947 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":69,"threads_started":1}
I20260619 08:56:14.282001 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:14.279620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51423 (local address 127.0.98.3:45917)
0619 08:56:14.279657 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:14.279659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:14.279738 (+ 79us) server_negotiation.cc:407] Connection header received
0619 08:56:14.279803 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:14.279803 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:14.279813 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:14.279857 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:56:14.279999 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:14.280518 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:14.281296 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:14.281588 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:14.281645 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:14.281767 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:56:14.281802 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:56:14.281844 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:14.281910 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 08:56:14.281930 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260619 08:56:14.282383 2689 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 08:56:15.806517 2119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:56:15.806627 2119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:56:15.807533 2119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5efeda2ae7b6428aada15b2780867507: Sending DeleteTablet for 3 replicas of tablet 05553009df464eb1967ad8ef3c48264e
I20260619 08:56:15.807798 2257 tablet_service.cc:1514] Processing DeleteTablet for tablet 05553009df464eb1967ad8ef3c48264e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51004
I20260619 08:56:15.807839 2397 tablet_service.cc:1514] Processing DeleteTablet for tablet 05553009df464eb1967ad8ef3c48264e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38508
I20260619 08:56:15.807945 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 2192
I20260619 08:56:15.807987 2537 tablet_service.cc:1514] Processing DeleteTablet for tablet 05553009df464eb1967ad8ef3c48264e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49136
I20260619 08:56:15.809391 2774 tablet_replica.cc:330] T 05553009df464eb1967ad8ef3c48264e P 1f909d146c124ed7849be361ca427112: stopping tablet replica
I20260619 08:56:15.809450 2774 raft_consensus.cc:2229] T 05553009df464eb1967ad8ef3c48264e P 1f909d146c124ed7849be361ca427112 [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:56:15.809547 2774 raft_consensus.cc:2258] T 05553009df464eb1967ad8ef3c48264e P 1f909d146c124ed7849be361ca427112 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:15.809655 2773 tablet_replica.cc:330] T 05553009df464eb1967ad8ef3c48264e P f2fad2d8cc29452d8cef0645665b4a8c: stopping tablet replica
I20260619 08:56:15.809708 2773 raft_consensus.cc:2229] T 05553009df464eb1967ad8ef3c48264e P f2fad2d8cc29452d8cef0645665b4a8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:56:15.809741 2773 raft_consensus.cc:2258] T 05553009df464eb1967ad8ef3c48264e P f2fad2d8cc29452d8cef0645665b4a8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:15.809801 2774 ts_tablet_manager.cc:1905] T 05553009df464eb1967ad8ef3c48264e P 1f909d146c124ed7849be361ca427112: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:15.809988 2773 ts_tablet_manager.cc:1905] T 05553009df464eb1967ad8ef3c48264e P f2fad2d8cc29452d8cef0645665b4a8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:15.811522 2774 ts_tablet_manager.cc:1918] T 05553009df464eb1967ad8ef3c48264e P 1f909d146c124ed7849be361ca427112: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 08:56:15.811582 2774 log.cc:1196] T 05553009df464eb1967ad8ef3c48264e P 1f909d146c124ed7849be361ca427112: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-2/wal/wals/05553009df464eb1967ad8ef3c48264e
I20260619 08:56:15.811691 2773 ts_tablet_manager.cc:1918] T 05553009df464eb1967ad8ef3c48264e P f2fad2d8cc29452d8cef0645665b4a8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 08:56:15.811753 2773 log.cc:1196] T 05553009df464eb1967ad8ef3c48264e P f2fad2d8cc29452d8cef0645665b4a8c: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859361540516-392-0/minicluster-data/ts-1/wal/wals/05553009df464eb1967ad8ef3c48264e
I20260619 08:56:15.811823 2774 ts_tablet_manager.cc:1939] T 05553009df464eb1967ad8ef3c48264e P 1f909d146c124ed7849be361ca427112: Deleting consensus metadata
I20260619 08:56:15.811990 2773 ts_tablet_manager.cc:1939] T 05553009df464eb1967ad8ef3c48264e P f2fad2d8cc29452d8cef0645665b4a8c: Deleting consensus metadata
I20260619 08:56:15.812153 2107 catalog_manager.cc:4841] TS 1f909d146c124ed7849be361ca427112 (127.0.98.3:45917): tablet 05553009df464eb1967ad8ef3c48264e (table test-table [id=cd4d92a0b511483eb385742f1bf8fe60]) successfully deleted
I20260619 08:56:15.812300 2107 catalog_manager.cc:4841] TS f2fad2d8cc29452d8cef0645665b4a8c (127.0.98.2:34599): tablet 05553009df464eb1967ad8ef3c48264e (table test-table [id=cd4d92a0b511483eb385742f1bf8fe60]) successfully deleted
W20260619 08:56:15.814702 2107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34365 (error 108)
I20260619 08:56:15.814776 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 2332
I20260619 08:56:15.816377 2776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:15.815573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34365 (local address 127.0.0.1:51018)
0619 08:56:15.816270 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:15.816329 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":57,"threads_started":1}
W20260619 08:56:15.816519 2107 catalog_manager.cc:4584] TS facf427fe9e541b781955151f2be5909 (127.0.98.1:34365): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 05553009df464eb1967ad8ef3c48264e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34365: connect: Connection refused (error 111)
I20260619 08:56:15.820187 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 2472
I20260619 08:56:15.825242 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 2088
2026-06-19T08:56:15Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5711 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:56:15 dist-test-slave-h4g5 krb5kdc[2779](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:56:15 dist-test-slave-h4g5 krb5kdc[2779](info): set up 2 sockets
Jun 19 08:56:15 dist-test-slave-h4g5 krb5kdc[2779](info): commencing operation
krb5kdc: starting...
W20260619 08:56:16.779620 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 08:56:16 dist-test-slave-h4g5 krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:56:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:56:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 08:56:16.895565 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46401
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46401
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:16.975389 2792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:16.975615 2792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:16.975646 2792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:16.976851 2792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:56:16.976898 2792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:16.976917 2792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:16.976934 2792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:56:16.976949 2792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:56:16.978536 2792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:16.978786 2792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:16.979004 2792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:16.983734 2800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:16.983779 2797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:16.983758 2798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:16.984001 2792 server_base.cc:1034] running on GCE node
I20260619 08:56:16.984256 2792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:16.984539 2792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:16.985716 2792 hybrid_clock.cc:648] HybridClock initialized: now 1781859376985695 us; error 33 us; skew 500 ppm
Jun 19 08:56:16 dist-test-slave-h4g5 krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:16.994315 2792 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 08:56:16.994822 2792 webserver.cc:460] Webserver started at http://127.0.98.62:33689/ using document root <none> and password file <none>
I20260619 08:56:16.995054 2792 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:16.995105 2792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:16.995237 2792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:16.996369 2792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "277fe8d3deaa418b83a0d154811b48c9"
format_stamp: "Formatted at 2026-06-19 08:56:16 on dist-test-slave-h4g5"
server_key: "4195ee1df4ccb4b1cf570b957e966e16"
server_key_iv: "7935199ab970f65416f0e8aaffcce5a4"
server_key_version: "encryptionkey@0"
I20260619 08:56:16.996704 2792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "277fe8d3deaa418b83a0d154811b48c9"
format_stamp: "Formatted at 2026-06-19 08:56:16 on dist-test-slave-h4g5"
server_key: "4195ee1df4ccb4b1cf570b957e966e16"
server_key_iv: "7935199ab970f65416f0e8aaffcce5a4"
server_key_version: "encryptionkey@0"
I20260619 08:56:16.998035 2792 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:56:16.998785 2807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:16.999048 2792 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:16.999143 2792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "277fe8d3deaa418b83a0d154811b48c9"
format_stamp: "Formatted at 2026-06-19 08:56:16 on dist-test-slave-h4g5"
server_key: "4195ee1df4ccb4b1cf570b957e966e16"
server_key_iv: "7935199ab970f65416f0e8aaffcce5a4"
server_key_version: "encryptionkey@0"
I20260619 08:56:16.999209 2792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:17.016552 2792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:17.018790 2792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:17.018947 2792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:17.022687 2792 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46401
I20260619 08:56:17.022737 2859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46401 every 8 connection(s)
I20260619 08:56:17.023216 2792 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:56:17.023895 2860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:17.026351 2860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 277fe8d3deaa418b83a0d154811b48c9: Bootstrap starting.
I20260619 08:56:17.026962 2860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 277fe8d3deaa418b83a0d154811b48c9: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:17.027225 2860 log.cc:824] T 00000000000000000000000000000000 P 277fe8d3deaa418b83a0d154811b48c9: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:17.027964 2860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 277fe8d3deaa418b83a0d154811b48c9: No bootstrap required, opened a new log
I20260619 08:56:17.029619 2860 raft_consensus.cc:348] T 00000000000000000000000000000000 P 277fe8d3deaa418b83a0d154811b48c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "277fe8d3deaa418b83a0d154811b48c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46401 } }
I20260619 08:56:17.029769 2860 raft_consensus.cc:374] T 00000000000000000000000000000000 P 277fe8d3deaa418b83a0d154811b48c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:17.029791 2860 raft_consensus.cc:729] T 00000000000000000000000000000000 P 277fe8d3deaa418b83a0d154811b48c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 277fe8d3deaa418b83a0d154811b48c9, State: Initialized, Role: FOLLOWER
I20260619 08:56:17.029898 2860 consensus_queue.cc:260] T 00000000000000000000000000000000 P 277fe8d3deaa418b83a0d154811b48c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "277fe8d3deaa418b83a0d154811b48c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46401 } }
I20260619 08:56:17.029951 2860 raft_consensus.cc:388] T 00000000000000000000000000000000 P 277fe8d3deaa418b83a0d154811b48c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:56:17.029978 2860 raft_consensus.cc:482] T 00000000000000000000000000000000 P 277fe8d3deaa418b83a0d154811b48c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:56:17.030018 2860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 277fe8d3deaa418b83a0d154811b48c9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:17.030536 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 2792
I20260619 08:56:17.030608 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 08:56:17.030756 392 external_mini_cluster.cc:1428] Setting key 6bbfc437dee69e9be57d21bf54bc443c
I20260619 08:56:17.031006 2860 raft_consensus.cc:504] T 00000000000000000000000000000000 P 277fe8d3deaa418b83a0d154811b48c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "277fe8d3deaa418b83a0d154811b48c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46401 } }
I20260619 08:56:17.031139 2860 leader_election.cc:302] T 00000000000000000000000000000000 P 277fe8d3deaa418b83a0d154811b48c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 277fe8d3deaa418b83a0d154811b48c9; no voters:
I20260619 08:56:17.031351 2860 leader_election.cc:288] T 00000000000000000000000000000000 P 277fe8d3deaa418b83a0d154811b48c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:56:17.031380 2864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 277fe8d3deaa418b83a0d154811b48c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:17.031562 2864 raft_consensus.cc:686] T 00000000000000000000000000000000 P 277fe8d3deaa418b83a0d154811b48c9 [term 1 LEADER]: Becoming Leader. State: Replica: 277fe8d3deaa418b83a0d154811b48c9, State: Running, Role: LEADER
I20260619 08:56:17.031713 2860 sys_catalog.cc:564] T 00000000000000000000000000000000 P 277fe8d3deaa418b83a0d154811b48c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:56:17.031718 2864 consensus_queue.cc:237] T 00000000000000000000000000000000 P 277fe8d3deaa418b83a0d154811b48c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "277fe8d3deaa418b83a0d154811b48c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46401 } }
I20260619 08:56:17.032239 2866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 277fe8d3deaa418b83a0d154811b48c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "277fe8d3deaa418b83a0d154811b48c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "277fe8d3deaa418b83a0d154811b48c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46401 } } }
I20260619 08:56:17.032251 2867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 277fe8d3deaa418b83a0d154811b48c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 277fe8d3deaa418b83a0d154811b48c9. Latest consensus state: current_term: 1 leader_uuid: "277fe8d3deaa418b83a0d154811b48c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "277fe8d3deaa418b83a0d154811b48c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46401 } } }
I20260619 08:56:17.032411 2866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 277fe8d3deaa418b83a0d154811b48c9 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:17.032425 2867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 277fe8d3deaa418b83a0d154811b48c9 [sys.catalog]: This master's current role is: LEADER
W20260619 08:56:17.033679 2881 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 277fe8d3deaa418b83a0d154811b48c9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 08:56:17.033757 2881 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 08:56:17.033901 2873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:56:17.034049 2873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:56:17.035574 2873 catalog_manager.cc:1347] Generated new cluster ID: d1b72ce99f9544a1b9dfe1e36760cc74
I20260619 08:56:17.035624 2873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 08:56:17 dist-test-slave-h4g5 krb5kdc[2779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:17.041939 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:17.031249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33188 (local address 127.0.98.62:46401)
0619 08:56:17.031469 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:17.031480 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:17.031511 (+ 31us) server_negotiation.cc:407] Connection header received
0619 08:56:17.031639 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:17.031647 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:17.031705 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:17.032694 (+ 989us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:17.032778 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:17.034147 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:17.035033 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:17.035647 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:17.035790 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:17.038468 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:17.038472 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:17.038472 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:17.038484 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:17.040374 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:17.040750 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:17.040752 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:17.040753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:17.040856 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:17.041047 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:17.041047 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:17.041048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:17.041344 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:17.041496 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:17.041590 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 08:56:17.041727 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":96,"threads_started":1}
I20260619 08:56:17.049546 2873 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:56:17.050297 2873 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:56:17.056444 2873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 277fe8d3deaa418b83a0d154811b48c9: Generated new TSK 0
I20260619 08:56:17.056659 2873 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 08:56:17.087410 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46401
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:17.167320 2886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:17.167544 2886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:17.167575 2886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:17.168774 2886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:17.168815 2886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:17.168848 2886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 08:56:17.170379 2886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:17.170620 2886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:17.170836 2886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:17.175807 2891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:17.175796 2892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:17.175819 2894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:17.175848 2886 server_base.cc:1034] running on GCE node
I20260619 08:56:17.176230 2886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:17.176461 2886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:17.177600 2886 hybrid_clock.cc:648] HybridClock initialized: now 1781859377177586 us; error 40 us; skew 500 ppm
Jun 19 08:56:17 dist-test-slave-h4g5 krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:17.179955 2886 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 08:56:17.180475 2886 webserver.cc:460] Webserver started at http://127.0.98.1:39235/ using document root <none> and password file <none>
I20260619 08:56:17.180688 2886 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:17.180735 2886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:17.180859 2886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:17.181944 2886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-0/data/instance:
uuid: "dd87f33c0fa442c2af203cb5e27fe7c7"
format_stamp: "Formatted at 2026-06-19 08:56:17 on dist-test-slave-h4g5"
server_key: "828bb3f1a5c715a89415d8ec8bdffc58"
server_key_iv: "dc0fb25db382eeea563cfbcee19d95b2"
server_key_version: "encryptionkey@0"
I20260619 08:56:17.182305 2886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dd87f33c0fa442c2af203cb5e27fe7c7"
format_stamp: "Formatted at 2026-06-19 08:56:17 on dist-test-slave-h4g5"
server_key: "828bb3f1a5c715a89415d8ec8bdffc58"
server_key_iv: "dc0fb25db382eeea563cfbcee19d95b2"
server_key_version: "encryptionkey@0"
I20260619 08:56:17.183610 2886 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:56:17.184314 2901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:17.184464 2886 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 08:56:17.184543 2886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-0/wal
uuid: "dd87f33c0fa442c2af203cb5e27fe7c7"
format_stamp: "Formatted at 2026-06-19 08:56:17 on dist-test-slave-h4g5"
server_key: "828bb3f1a5c715a89415d8ec8bdffc58"
server_key_iv: "dc0fb25db382eeea563cfbcee19d95b2"
server_key_version: "encryptionkey@0"
I20260619 08:56:17.184604 2886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:17.204249 2886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:17.206133 2886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:17.206278 2886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:17.206496 2886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:17.206786 2886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:17.206820 2886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:17.206847 2886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:17.206868 2886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:17.212488 2886 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39201
I20260619 08:56:17.212528 3014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39201 every 8 connection(s)
I20260619 08:56:17.212836 2886 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 08:56:17 dist-test-slave-h4g5 krb5kdc[2779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:17.221889 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:17.213492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45107 (local address 127.0.98.62:46401)
0619 08:56:17.213550 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:17.213552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:17.213771 (+ 219us) server_negotiation.cc:407] Connection header received
0619 08:56:17.214656 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:17.214657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:17.214673 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:17.214736 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:17.215382 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:17.215880 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:17.216928 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:17.217133 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:17.217200 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:17.219184 (+ 1984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:17.219186 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:17.219186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:17.219193 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:17.220625 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:17.221024 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:17.221026 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:17.221026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:17.221075 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:17.221287 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:17.221288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:17.221288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:17.221403 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:17.221459 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:17.221718 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 08:56:17.221798 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 08:56:17.222069 3017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:17.213473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46401 (local address 127.0.98.1:45107)
0619 08:56:17.213621 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:17.213664 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:17.214558 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:17.214833 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:17.214836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:17.215082 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:17.215304 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:17.215309 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:17.215959 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:17.215960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:17.216845 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:17.216847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:17.217222 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:17.217223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:17.217324 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:17.217740 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:17.217748 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:17.219090 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:17.220703 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:17.220711 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:17.220713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:17.220956 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:17.221128 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:17.221129 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:17.221129 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:17.221233 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:17.221458 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:17.221465 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:17.221603 (+ 138us) client_negotiation.cc:769] Sending connection context
0619 08:56:17.221692 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 08:56:17.221857 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260619 08:56:17.222543 3015 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46401
I20260619 08:56:17.222659 3015 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:17.222682 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 2886
I20260619 08:56:17.222761 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-0/wal/instance
I20260619 08:56:17.222857 3015 heartbeater.cc:511] Master 127.0.98.62:46401 requested a full tablet report, sending...
I20260619 08:56:17.222959 392 external_mini_cluster.cc:1428] Setting key a8a199db8fed3f82be3ff2c6a1f5d672
I20260619 08:56:17.223415 2824 ts_manager.cc:194] Registered new tserver with Master: dd87f33c0fa442c2af203cb5e27fe7c7 (127.0.98.1:39201)
I20260619 08:56:17.224035 2824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45107
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 08:56:17.246529 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46401
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:17.329268 3020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:17.329481 3020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:17.329512 3020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:17.330701 3020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:17.330742 3020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:17.330777 3020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 08:56:17.332420 3020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:17.332700 3020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:17.332911 3020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:17.337579 3026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:17.337616 3025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:17.337666 3028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:17.337769 3020 server_base.cc:1034] running on GCE node
I20260619 08:56:17.337975 3020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:17.338248 3020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:17.339433 3020 hybrid_clock.cc:648] HybridClock initialized: now 1781859377339410 us; error 36 us; skew 500 ppm
Jun 19 08:56:17 dist-test-slave-h4g5 krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:17.342103 3020 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 08:56:17.342566 3020 webserver.cc:460] Webserver started at http://127.0.98.2:44129/ using document root <none> and password file <none>
I20260619 08:56:17.342764 3020 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:17.342808 3020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:17.342904 3020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:17.343919 3020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-1/data/instance:
uuid: "ba8f4a725d224dd4ba9b815c1f530d13"
format_stamp: "Formatted at 2026-06-19 08:56:17 on dist-test-slave-h4g5"
server_key: "2593ab415690290714642ea1a0d8fce1"
server_key_iv: "5f5adbd985dc5bc6d665b5852828ef49"
server_key_version: "encryptionkey@0"
I20260619 08:56:17.344295 3020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ba8f4a725d224dd4ba9b815c1f530d13"
format_stamp: "Formatted at 2026-06-19 08:56:17 on dist-test-slave-h4g5"
server_key: "2593ab415690290714642ea1a0d8fce1"
server_key_iv: "5f5adbd985dc5bc6d665b5852828ef49"
server_key_version: "encryptionkey@0"
I20260619 08:56:17.345589 3020 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:17.346266 3035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:17.346444 3020 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:17.346526 3020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-1/wal
uuid: "ba8f4a725d224dd4ba9b815c1f530d13"
format_stamp: "Formatted at 2026-06-19 08:56:17 on dist-test-slave-h4g5"
server_key: "2593ab415690290714642ea1a0d8fce1"
server_key_iv: "5f5adbd985dc5bc6d665b5852828ef49"
server_key_version: "encryptionkey@0"
I20260619 08:56:17.346598 3020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:17.362917 3020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:17.364948 3020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:17.365072 3020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:17.365305 3020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:17.365597 3020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:17.365630 3020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:17.365660 3020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:17.365677 3020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:17.371039 3020 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40191
I20260619 08:56:17.371099 3148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40191 every 8 connection(s)
I20260619 08:56:17.371434 3020 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 08:56:17 dist-test-slave-h4g5 krb5kdc[2779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:17.380450 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:17.372132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37921 (local address 127.0.98.62:46401)
0619 08:56:17.372178 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:17.372180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:17.372404 (+ 224us) server_negotiation.cc:407] Connection header received
0619 08:56:17.373302 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:17.373303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:17.373313 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:17.373362 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:17.374156 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:17.374772 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:17.375741 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:17.375928 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:17.375983 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:17.377982 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:17.377983 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:17.377984 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:17.377991 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:17.379149 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:17.379529 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:17.379531 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:17.379531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:17.379580 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:17.379787 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:17.379788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:17.379788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:17.379893 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:17.379948 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:17.380261 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 08:56:17.380358 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 08:56:17.380606 3151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:17.372109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46401 (local address 127.0.98.2:37921)
0619 08:56:17.372257 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:17.372292 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:17.373209 (+ 917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:17.373444 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:17.373447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:17.373786 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:17.374056 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:17.374061 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:17.374847 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:17.374848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:17.375648 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:17.375650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:17.375990 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:17.375990 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:17.376097 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:17.376583 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:17.376592 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:17.377887 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:17.379247 (+ 1360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:17.379253 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:17.379256 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:17.379464 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:17.379633 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:17.379634 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:17.379634 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:17.379734 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:17.379946 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:17.379952 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:17.380119 (+ 167us) client_negotiation.cc:769] Sending connection context
0619 08:56:17.380235 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 08:56:17.380393 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
I20260619 08:56:17.381125 3149 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46401
I20260619 08:56:17.381242 3149 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:17.381333 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 3020
I20260619 08:56:17.381415 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-1/wal/instance
I20260619 08:56:17.381479 3149 heartbeater.cc:511] Master 127.0.98.62:46401 requested a full tablet report, sending...
I20260619 08:56:17.381621 392 external_mini_cluster.cc:1428] Setting key 0fb9816b7cba032d3e4e048b8af2d6cb
I20260619 08:56:17.381881 2824 ts_manager.cc:194] Registered new tserver with Master: ba8f4a725d224dd4ba9b815c1f530d13 (127.0.98.2:40191)
I20260619 08:56:17.382316 2824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37921
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 08:56:17.407502 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46401
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:17.487756 3154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:17.488005 3154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:17.488031 3154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:17.489317 3154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:17.489369 3154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:17.489398 3154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 08:56:17.491014 3154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:17.491230 3154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:17.491417 3154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:17.496263 3159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:17.496331 3160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:17.496289 3162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:17.496801 3154 server_base.cc:1034] running on GCE node
I20260619 08:56:17.497013 3154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:17.497243 3154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:17.498397 3154 hybrid_clock.cc:648] HybridClock initialized: now 1781859377498378 us; error 34 us; skew 500 ppm
Jun 19 08:56:17 dist-test-slave-h4g5 krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:17.500955 3154 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 08:56:17.501417 3154 webserver.cc:460] Webserver started at http://127.0.98.3:45441/ using document root <none> and password file <none>
I20260619 08:56:17.501648 3154 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:17.501698 3154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:17.501820 3154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:17.502880 3154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-2/data/instance:
uuid: "fa4dd14397b7477f9f0437d8f5da732d"
format_stamp: "Formatted at 2026-06-19 08:56:17 on dist-test-slave-h4g5"
server_key: "1c381b73bdd816842b333a82393ea89f"
server_key_iv: "ae51eee0a051359fd7fee3f00aaec138"
server_key_version: "encryptionkey@0"
I20260619 08:56:17.503206 3154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fa4dd14397b7477f9f0437d8f5da732d"
format_stamp: "Formatted at 2026-06-19 08:56:17 on dist-test-slave-h4g5"
server_key: "1c381b73bdd816842b333a82393ea89f"
server_key_iv: "ae51eee0a051359fd7fee3f00aaec138"
server_key_version: "encryptionkey@0"
I20260619 08:56:17.504472 3154 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260619 08:56:17.505190 3169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:17.505342 3154 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:17.505427 3154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-2/wal
uuid: "fa4dd14397b7477f9f0437d8f5da732d"
format_stamp: "Formatted at 2026-06-19 08:56:17 on dist-test-slave-h4g5"
server_key: "1c381b73bdd816842b333a82393ea89f"
server_key_iv: "ae51eee0a051359fd7fee3f00aaec138"
server_key_version: "encryptionkey@0"
I20260619 08:56:17.505487 3154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:17.518925 3154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:17.521106 3154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:17.521246 3154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:17.521484 3154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:17.521772 3154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:17.521806 3154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:17.521834 3154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:17.521852 3154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:17.527194 3154 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38221
I20260619 08:56:17.527292 3282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38221 every 8 connection(s)
I20260619 08:56:17.527570 3154 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
I20260619 08:56:17.532383 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 3154
I20260619 08:56:17.532477 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-2/wal/instance
I20260619 08:56:17.532667 392 external_mini_cluster.cc:1428] Setting key 3612315997f23cae011910a8131482b5
Jun 19 08:56:17 dist-test-slave-h4g5 krb5kdc[2779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:17.537102 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:17.528347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43747 (local address 127.0.98.62:46401)
0619 08:56:17.528424 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:17.528426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:17.528600 (+ 174us) server_negotiation.cc:407] Connection header received
0619 08:56:17.529605 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:17.529606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:17.529617 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:17.529667 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:17.530385 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:17.530894 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:17.532021 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:17.532426 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:17.532497 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:17.534790 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:17.534791 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:17.534791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:17.534798 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:17.535850 (+ 1052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:17.536259 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:17.536260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:17.536260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:17.536309 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:17.536555 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:17.536557 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:17.536557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:17.536683 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:17.536758 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:17.536963 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 08:56:17.537025 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 08:56:17.537287 3285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:17.528306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46401 (local address 127.0.98.3:43747)
0619 08:56:17.528476 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:17.528516 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:17.529503 (+ 987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:17.529746 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:17.529750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:17.530026 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:17.530246 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:17.530251 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:17.531000 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:17.531001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:17.531935 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:17.531938 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:17.532533 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:17.532535 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:17.532696 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:17.533285 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:17.533296 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:17.534708 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:17.535914 (+ 1206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:17.535920 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:17.535922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:17.536202 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:17.536356 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:17.536357 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:17.536358 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:17.536473 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:17.536745 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:17.536749 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:17.536872 (+ 123us) client_negotiation.cc:769] Sending connection context
0619 08:56:17.536949 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 08:56:17.537070 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":71,"threads_started":1}
I20260619 08:56:17.537721 3283 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46401
I20260619 08:56:17.537803 3283 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:17.537957 3283 heartbeater.cc:511] Master 127.0.98.62:46401 requested a full tablet report, sending...
I20260619 08:56:17.538281 2824 ts_manager.cc:194] Registered new tserver with Master: fa4dd14397b7477f9f0437d8f5da732d (127.0.98.3:38221)
I20260619 08:56:17.538631 2824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43747
I20260619 08:56:17.544452 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 08:56:17.555349 2865 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:17.554363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33200 (local address 127.0.98.62:46401)
0619 08:56:17.554474 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:17.554477 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:17.554558 (+ 81us) server_negotiation.cc:407] Connection header received
0619 08:56:17.554634 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:17.554635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:17.554646 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:17.554691 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:17.555267 (+ 576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33200: BlockingRecv error: recv got EOF from 127.0.0.1:33200 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 08:56:17.556237 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 2886
I20260619 08:56:17.560192 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 3020
I20260619 08:56:17.563993 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 3154
I20260619 08:56:17.567577 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 2792
2026-06-19T08:56:17Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1741 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:56:17 dist-test-slave-h4g5 krb5kdc[3295](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:56:17 dist-test-slave-h4g5 krb5kdc[3295](info): set up 2 sockets
Jun 19 08:56:17 dist-test-slave-h4g5 krb5kdc[3295](info): commencing operation
krb5kdc: starting...
W20260619 08:56:18.531744 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 08:56:18 dist-test-slave-h4g5 krb5kdc[3295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:56:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:56:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 08:56:18.647099 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37639
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:18.728729 3308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:18.728964 3308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:18.728996 3308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:18.730203 3308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:56:18.730247 3308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:18.730264 3308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:18.730281 3308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:56:18.730295 3308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:56:18.731812 3308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:18.732065 3308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:18.732298 3308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:18.736886 3313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:18.736898 3316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:18.736898 3314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:18.736963 3308 server_base.cc:1034] running on GCE node
I20260619 08:56:18.737330 3308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:18.737641 3308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:18.738793 3308 hybrid_clock.cc:648] HybridClock initialized: now 1781859378738783 us; error 43 us; skew 500 ppm
Jun 19 08:56:18 dist-test-slave-h4g5 krb5kdc[3295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:18.741170 3308 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 08:56:18.741591 3308 webserver.cc:460] Webserver started at http://127.0.98.62:36401/ using document root <none> and password file <none>
I20260619 08:56:18.741799 3308 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:18.741847 3308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:18.741976 3308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:18.743017 3308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "a3387eaf79034a27af64d1e9289be939"
format_stamp: "Formatted at 2026-06-19 08:56:18 on dist-test-slave-h4g5"
server_key: "0d03c7b923c3403622f8d872acb2dc23"
server_key_iv: "b9c223bdf2c9e456344fcc4fb6ed59ba"
server_key_version: "encryptionkey@0"
I20260619 08:56:18.743335 3308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "a3387eaf79034a27af64d1e9289be939"
format_stamp: "Formatted at 2026-06-19 08:56:18 on dist-test-slave-h4g5"
server_key: "0d03c7b923c3403622f8d872acb2dc23"
server_key_iv: "b9c223bdf2c9e456344fcc4fb6ed59ba"
server_key_version: "encryptionkey@0"
I20260619 08:56:18.744639 3308 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:56:18.745368 3323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:18.745581 3308 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:56:18.745666 3308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "a3387eaf79034a27af64d1e9289be939"
format_stamp: "Formatted at 2026-06-19 08:56:18 on dist-test-slave-h4g5"
server_key: "0d03c7b923c3403622f8d872acb2dc23"
server_key_iv: "b9c223bdf2c9e456344fcc4fb6ed59ba"
server_key_version: "encryptionkey@0"
I20260619 08:56:18.745728 3308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:18.772277 3308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:18.774416 3308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:18.774564 3308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:18.778226 3308 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37639
I20260619 08:56:18.778272 3375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37639 every 8 connection(s)
I20260619 08:56:18.778600 3308 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:56:18.779424 3376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:18.781760 3376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3387eaf79034a27af64d1e9289be939: Bootstrap starting.
I20260619 08:56:18.782289 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 3308
I20260619 08:56:18.782305 3376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3387eaf79034a27af64d1e9289be939: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:18.782363 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 08:56:18.782512 392 external_mini_cluster.cc:1428] Setting key 2729ed9309e96a1c08d2f2588698f609
I20260619 08:56:18.782580 3376 log.cc:824] T 00000000000000000000000000000000 P a3387eaf79034a27af64d1e9289be939: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:18.783661 3376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3387eaf79034a27af64d1e9289be939: No bootstrap required, opened a new log
I20260619 08:56:18.785449 3376 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3387eaf79034a27af64d1e9289be939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3387eaf79034a27af64d1e9289be939" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37639 } }
I20260619 08:56:18.785583 3376 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3387eaf79034a27af64d1e9289be939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:18.785614 3376 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3387eaf79034a27af64d1e9289be939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3387eaf79034a27af64d1e9289be939, State: Initialized, Role: FOLLOWER
I20260619 08:56:18.785735 3376 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3387eaf79034a27af64d1e9289be939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3387eaf79034a27af64d1e9289be939" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37639 } }
I20260619 08:56:18.785805 3376 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3387eaf79034a27af64d1e9289be939 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:56:18.785845 3376 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3387eaf79034a27af64d1e9289be939 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:56:18.785892 3376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3387eaf79034a27af64d1e9289be939 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:18.786409 3376 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3387eaf79034a27af64d1e9289be939 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3387eaf79034a27af64d1e9289be939" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37639 } }
I20260619 08:56:18.786509 3376 leader_election.cc:302] T 00000000000000000000000000000000 P a3387eaf79034a27af64d1e9289be939 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3387eaf79034a27af64d1e9289be939; no voters:
I20260619 08:56:18.786669 3376 leader_election.cc:288] T 00000000000000000000000000000000 P a3387eaf79034a27af64d1e9289be939 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:56:18.786839 3382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3387eaf79034a27af64d1e9289be939 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:18.786881 3376 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3387eaf79034a27af64d1e9289be939 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:56:18.786983 3382 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3387eaf79034a27af64d1e9289be939 [term 1 LEADER]: Becoming Leader. State: Replica: a3387eaf79034a27af64d1e9289be939, State: Running, Role: LEADER
I20260619 08:56:18.787109 3382 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3387eaf79034a27af64d1e9289be939 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3387eaf79034a27af64d1e9289be939" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37639 } }
I20260619 08:56:18.787436 3381 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3387eaf79034a27af64d1e9289be939 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3387eaf79034a27af64d1e9289be939" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3387eaf79034a27af64d1e9289be939" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37639 } } }
I20260619 08:56:18.787858 3381 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3387eaf79034a27af64d1e9289be939 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:18.787448 3383 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3387eaf79034a27af64d1e9289be939 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3387eaf79034a27af64d1e9289be939. Latest consensus state: current_term: 1 leader_uuid: "a3387eaf79034a27af64d1e9289be939" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3387eaf79034a27af64d1e9289be939" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37639 } } }
I20260619 08:56:18.788043 3383 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3387eaf79034a27af64d1e9289be939 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:18.788285 3390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:56:18.788738 3390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 08:56:18 dist-test-slave-h4g5 krb5kdc[3295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:18.790134 3390 catalog_manager.cc:1347] Generated new cluster ID: eed2df6bc2e54b32b87820b7626a98d3
I20260619 08:56:18.790181 3390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:56:18.800352 3379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:18.782981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58084 (local address 127.0.98.62:37639)
0619 08:56:18.783164 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:18.783174 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:18.783209 (+ 35us) server_negotiation.cc:407] Connection header received
0619 08:56:18.783646 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:18.783653 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:18.783705 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:18.783849 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:18.784455 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:18.785320 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:18.786185 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:18.786620 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:18.786720 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:18.790014 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:18.790017 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:18.790018 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:18.790029 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:18.798641 (+ 8612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:18.799112 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:18.799115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:18.799115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:18.799221 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:18.799402 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:18.799403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:18.799404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:18.799710 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:18.799845 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:18.799953 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 08:56:18.800120 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":78,"threads_started":1}
I20260619 08:56:18.801409 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 08:56:18.802300 3390 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:56:18.802774 3390 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:56:18.821599 3390 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3387eaf79034a27af64d1e9289be939: Generated new TSK 0
I20260619 08:56:18.821858 3390 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:56:18.830008 3379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:18.825786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58086 (local address 127.0.98.62:37639)
0619 08:56:18.825839 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:18.825841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:18.825981 (+ 140us) server_negotiation.cc:407] Connection header received
0619 08:56:18.826014 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:18.826015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:18.826025 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:18.826081 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:18.826558 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:18.827200 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:18.828389 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:18.828620 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:18.828682 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:18.829456 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:18.829457 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:18.829458 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:56:18.829697 (+ 239us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:18.829720 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:56:18.829727 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:18.829781 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:18.829878 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 08:56:18.829909 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 08:56:18.832895 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 3308
2026-06-19T08:56:18Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1264 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:56:18 dist-test-slave-h4g5 krb5kdc[3409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:56:18 dist-test-slave-h4g5 krb5kdc[3409](info): set up 2 sockets
Jun 19 08:56:18 dist-test-slave-h4g5 krb5kdc[3409](info): commencing operation
krb5kdc: starting...
W20260619 08:56:19.791759 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 08:56:19 dist-test-slave-h4g5 krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:56:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:56:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 08:56:19.906862 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35901
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:19.992538 3422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:19.992810 3422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:19.992846 3422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:19.994549 3422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:56:19.994603 3422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:19.994623 3422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:19.994640 3422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:56:19.994668 3422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:56:19.996861 3422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:19.997159 3422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:19.997426 3422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:20.002569 3427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:20.002633 3428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:20.002843 3422 server_base.cc:1034] running on GCE node
W20260619 08:56:20.003039 3430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:20.003333 3422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:20.003660 3422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:20.004817 3422 hybrid_clock.cc:648] HybridClock initialized: now 1781859380004802 us; error 33 us; skew 500 ppm
Jun 19 08:56:20 dist-test-slave-h4g5 krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:20.007319 3422 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 08:56:20.007783 3422 webserver.cc:460] Webserver started at http://127.0.98.62:37991/ using document root <none> and password file <none>
I20260619 08:56:20.007988 3422 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:20.008035 3422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:20.008193 3422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:20.009246 3422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "902a2fa5b4f74de0bafd83c4ba66ea00"
format_stamp: "Formatted at 2026-06-19 08:56:20 on dist-test-slave-h4g5"
server_key: "074ae635dd20f4be173114c7ea79cc50"
server_key_iv: "8ce071ce95575051de4a13885d39f2f7"
server_key_version: "encryptionkey@0"
I20260619 08:56:20.009574 3422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "902a2fa5b4f74de0bafd83c4ba66ea00"
format_stamp: "Formatted at 2026-06-19 08:56:20 on dist-test-slave-h4g5"
server_key: "074ae635dd20f4be173114c7ea79cc50"
server_key_iv: "8ce071ce95575051de4a13885d39f2f7"
server_key_version: "encryptionkey@0"
I20260619 08:56:20.010826 3422 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:56:20.011643 3437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:20.011848 3422 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:20.011937 3422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "902a2fa5b4f74de0bafd83c4ba66ea00"
format_stamp: "Formatted at 2026-06-19 08:56:20 on dist-test-slave-h4g5"
server_key: "074ae635dd20f4be173114c7ea79cc50"
server_key_iv: "8ce071ce95575051de4a13885d39f2f7"
server_key_version: "encryptionkey@0"
I20260619 08:56:20.012007 3422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:20.028928 3422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:20.031045 3422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:20.031203 3422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:20.034972 3422 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35901
I20260619 08:56:20.035015 3489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35901 every 8 connection(s)
I20260619 08:56:20.035336 3422 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:56:20.036342 3490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:20.038545 3490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 902a2fa5b4f74de0bafd83c4ba66ea00: Bootstrap starting.
I20260619 08:56:20.039076 3490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 902a2fa5b4f74de0bafd83c4ba66ea00: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:20.039291 3490 log.cc:824] T 00000000000000000000000000000000 P 902a2fa5b4f74de0bafd83c4ba66ea00: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:20.039922 3490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 902a2fa5b4f74de0bafd83c4ba66ea00: No bootstrap required, opened a new log
I20260619 08:56:20.041531 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 3422
I20260619 08:56:20.041630 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 08:56:20.041558 3490 raft_consensus.cc:348] T 00000000000000000000000000000000 P 902a2fa5b4f74de0bafd83c4ba66ea00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902a2fa5b4f74de0bafd83c4ba66ea00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35901 } }
I20260619 08:56:20.041790 3490 raft_consensus.cc:374] T 00000000000000000000000000000000 P 902a2fa5b4f74de0bafd83c4ba66ea00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:20.041812 392 external_mini_cluster.cc:1428] Setting key 2d60cc1ff70ade943d1b3eedc053e67a
I20260619 08:56:20.041831 3490 raft_consensus.cc:729] T 00000000000000000000000000000000 P 902a2fa5b4f74de0bafd83c4ba66ea00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 902a2fa5b4f74de0bafd83c4ba66ea00, State: Initialized, Role: FOLLOWER
I20260619 08:56:20.041977 3490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 902a2fa5b4f74de0bafd83c4ba66ea00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902a2fa5b4f74de0bafd83c4ba66ea00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35901 } }
I20260619 08:56:20.042045 3490 raft_consensus.cc:388] T 00000000000000000000000000000000 P 902a2fa5b4f74de0bafd83c4ba66ea00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:56:20.042069 3490 raft_consensus.cc:482] T 00000000000000000000000000000000 P 902a2fa5b4f74de0bafd83c4ba66ea00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:56:20.042116 3490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 902a2fa5b4f74de0bafd83c4ba66ea00 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:20.042946 3490 raft_consensus.cc:504] T 00000000000000000000000000000000 P 902a2fa5b4f74de0bafd83c4ba66ea00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902a2fa5b4f74de0bafd83c4ba66ea00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35901 } }
I20260619 08:56:20.043037 3490 leader_election.cc:302] T 00000000000000000000000000000000 P 902a2fa5b4f74de0bafd83c4ba66ea00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 902a2fa5b4f74de0bafd83c4ba66ea00; no voters:
I20260619 08:56:20.043182 3490 leader_election.cc:288] T 00000000000000000000000000000000 P 902a2fa5b4f74de0bafd83c4ba66ea00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:56:20.043211 3495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 902a2fa5b4f74de0bafd83c4ba66ea00 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:20.043303 3495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 902a2fa5b4f74de0bafd83c4ba66ea00 [term 1 LEADER]: Becoming Leader. State: Replica: 902a2fa5b4f74de0bafd83c4ba66ea00, State: Running, Role: LEADER
I20260619 08:56:20.043432 3490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 902a2fa5b4f74de0bafd83c4ba66ea00 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:56:20.043439 3495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 902a2fa5b4f74de0bafd83c4ba66ea00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902a2fa5b4f74de0bafd83c4ba66ea00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35901 } }
I20260619 08:56:20.043900 3497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 902a2fa5b4f74de0bafd83c4ba66ea00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 902a2fa5b4f74de0bafd83c4ba66ea00. Latest consensus state: current_term: 1 leader_uuid: "902a2fa5b4f74de0bafd83c4ba66ea00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902a2fa5b4f74de0bafd83c4ba66ea00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35901 } } }
I20260619 08:56:20.043915 3496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 902a2fa5b4f74de0bafd83c4ba66ea00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "902a2fa5b4f74de0bafd83c4ba66ea00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902a2fa5b4f74de0bafd83c4ba66ea00" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35901 } } }
I20260619 08:56:20.044075 3496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 902a2fa5b4f74de0bafd83c4ba66ea00 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:20.044063 3497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 902a2fa5b4f74de0bafd83c4ba66ea00 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:20.044622 3504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:56:20.045249 3504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:56:20.046712 3504 catalog_manager.cc:1347] Generated new cluster ID: a99da1b242be426685d6e3efd24d97ac
I20260619 08:56:20.046764 3504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 08:56:20 dist-test-slave-h4g5 krb5kdc[3409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:20.053473 3494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:20.042255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57668 (local address 127.0.98.62:35901)
0619 08:56:20.042414 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:20.042424 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:20.042476 (+ 52us) server_negotiation.cc:407] Connection header received
0619 08:56:20.042586 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:20.042591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:20.042633 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:20.042740 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:20.043392 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:20.044716 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:20.045770 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:20.046352 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:20.046487 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:20.050136 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:20.050140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:20.050140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:20.050151 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:20.052074 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:20.052411 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:20.052415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:20.052415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:20.052508 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:20.052688 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:20.052689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:20.052689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:20.052917 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:20.053072 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:20.053135 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 08:56:20.053271 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":77,"threads_started":1}
I20260619 08:56:20.062996 3504 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:56:20.063587 3504 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:56:20.069473 3504 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 902a2fa5b4f74de0bafd83c4ba66ea00: Generated new TSK 0
I20260619 08:56:20.069712 3504 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 08:56:20.096964 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35901
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:20.178499 3516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:20.178725 3516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:20.178758 3516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:20.179932 3516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:20.179975 3516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:20.180011 3516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 08:56:20.181490 3516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:20.181739 3516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:20.181944 3516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:20.186599 3522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:20.186596 3521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:20.186596 3524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:20.186863 3516 server_base.cc:1034] running on GCE node
I20260619 08:56:20.187122 3516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:20.187451 3516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:20.188663 3516 hybrid_clock.cc:648] HybridClock initialized: now 1781859380188640 us; error 32 us; skew 500 ppm
Jun 19 08:56:20 dist-test-slave-h4g5 krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:20.191416 3516 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 08:56:20.191965 3516 webserver.cc:460] Webserver started at http://127.0.98.1:43043/ using document root <none> and password file <none>
I20260619 08:56:20.192229 3516 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:20.192289 3516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:20.192440 3516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:20.193569 3516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-0/data/instance:
uuid: "0c846932a8444d7ca7fb34533cb82868"
format_stamp: "Formatted at 2026-06-19 08:56:20 on dist-test-slave-h4g5"
server_key: "0674028cd7d3ce4d171ae5eeb46b672f"
server_key_iv: "bd6dc006eb24c8277bf2604d613c758f"
server_key_version: "encryptionkey@0"
I20260619 08:56:20.193953 3516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0c846932a8444d7ca7fb34533cb82868"
format_stamp: "Formatted at 2026-06-19 08:56:20 on dist-test-slave-h4g5"
server_key: "0674028cd7d3ce4d171ae5eeb46b672f"
server_key_iv: "bd6dc006eb24c8277bf2604d613c758f"
server_key_version: "encryptionkey@0"
I20260619 08:56:20.195415 3516 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:56:20.196456 3531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:20.196689 3516 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:20.196789 3516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-0/wal
uuid: "0c846932a8444d7ca7fb34533cb82868"
format_stamp: "Formatted at 2026-06-19 08:56:20 on dist-test-slave-h4g5"
server_key: "0674028cd7d3ce4d171ae5eeb46b672f"
server_key_iv: "bd6dc006eb24c8277bf2604d613c758f"
server_key_version: "encryptionkey@0"
I20260619 08:56:20.196870 3516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:20.213220 3516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:20.215169 3516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:20.215309 3516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:20.215559 3516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:20.215919 3516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:20.215955 3516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:20.215984 3516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:20.216012 3516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:20.221916 3516 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33627
I20260619 08:56:20.221971 3644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33627 every 8 connection(s)
I20260619 08:56:20.222317 3516 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 08:56:20 dist-test-slave-h4g5 krb5kdc[3409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:20.232146 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 3516
I20260619 08:56:20.232239 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-0/wal/instance
I20260619 08:56:20.232429 392 external_mini_cluster.cc:1428] Setting key 2c5e28a6fdf9e4673d30cfc49e414d05
I20260619 08:56:20.232577 3494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:20.223101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40053 (local address 127.0.98.62:35901)
0619 08:56:20.223163 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:20.223165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:20.223340 (+ 175us) server_negotiation.cc:407] Connection header received
0619 08:56:20.223907 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:20.223908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:20.223923 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:20.223993 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:20.225046 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:20.225585 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:20.226670 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:20.226878 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:20.226944 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:20.229707 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:20.229709 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:20.229710 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:20.229717 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:20.231051 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:20.231500 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:20.231501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:20.231501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:20.231587 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:20.231828 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:20.231828 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:20.231829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:20.231968 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:20.232063 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:20.232388 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 08:56:20.232489 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 08:56:20.232805 3647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:20.223035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35901 (local address 127.0.98.1:40053)
0619 08:56:20.223203 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:20.223247 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:20.223768 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:20.224066 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:20.224069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:20.224719 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:20.224928 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:20.224934 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:20.225722 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:20.225723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:20.226564 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:20.226567 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:20.226999 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:20.227000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:20.227148 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:20.227793 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:20.227803 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:20.229602 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:20.231174 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:20.231182 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:20.231184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:20.231412 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:20.231654 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:20.231655 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:20.231655 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:20.231776 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:20.232054 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:20.232059 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:20.232257 (+ 198us) client_negotiation.cc:769] Sending connection context
0619 08:56:20.232357 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 08:56:20.232509 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":74,"threads_started":1}
I20260619 08:56:20.233268 3645 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35901
I20260619 08:56:20.233372 3645 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:20.233569 3645 heartbeater.cc:511] Master 127.0.98.62:35901 requested a full tablet report, sending...
I20260619 08:56:20.234125 3454 ts_manager.cc:194] Registered new tserver with Master: 0c846932a8444d7ca7fb34533cb82868 (127.0.98.1:33627)
I20260619 08:56:20.234781 3454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40053
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 08:56:20.257059 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35901
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:20.337671 3650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:20.337885 3650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:20.337908 3650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:20.339073 3650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:20.339120 3650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:20.339145 3650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 08:56:20.340591 3650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:20.340821 3650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:20.341034 3650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:20.346098 3658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:20.346110 3656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:20.346119 3655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:20.346350 3650 server_base.cc:1034] running on GCE node
I20260619 08:56:20.346551 3650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:20.346791 3650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:20.347947 3650 hybrid_clock.cc:648] HybridClock initialized: now 1781859380347938 us; error 32 us; skew 500 ppm
Jun 19 08:56:20 dist-test-slave-h4g5 krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:20.350795 3650 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 08:56:20.351250 3650 webserver.cc:460] Webserver started at http://127.0.98.2:43201/ using document root <none> and password file <none>
I20260619 08:56:20.351455 3650 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:20.351502 3650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:20.351629 3650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:20.352727 3650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-1/data/instance:
uuid: "6f602abaeacf4a95bc118acc0b665fc5"
format_stamp: "Formatted at 2026-06-19 08:56:20 on dist-test-slave-h4g5"
server_key: "2e6f102792a400a67feee21f8f6bf386"
server_key_iv: "2f2f5cf9ae44cbfe21d503d0c207d63f"
server_key_version: "encryptionkey@0"
I20260619 08:56:20.353056 3650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6f602abaeacf4a95bc118acc0b665fc5"
format_stamp: "Formatted at 2026-06-19 08:56:20 on dist-test-slave-h4g5"
server_key: "2e6f102792a400a67feee21f8f6bf386"
server_key_iv: "2f2f5cf9ae44cbfe21d503d0c207d63f"
server_key_version: "encryptionkey@0"
I20260619 08:56:20.354281 3650 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:56:20.355033 3665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:20.355203 3650 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:20.355283 3650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-1/wal
uuid: "6f602abaeacf4a95bc118acc0b665fc5"
format_stamp: "Formatted at 2026-06-19 08:56:20 on dist-test-slave-h4g5"
server_key: "2e6f102792a400a67feee21f8f6bf386"
server_key_iv: "2f2f5cf9ae44cbfe21d503d0c207d63f"
server_key_version: "encryptionkey@0"
I20260619 08:56:20.355348 3650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:20.361620 3650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:20.363353 3650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:20.363482 3650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:20.363687 3650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:20.363971 3650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:20.364003 3650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:20.364030 3650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:20.364053 3650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:20.369567 3650 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40227
I20260619 08:56:20.369626 3778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40227 every 8 connection(s)
I20260619 08:56:20.369918 3650 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
I20260619 08:56:20.371507 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 3650
I20260619 08:56:20.371611 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-1/wal/instance
I20260619 08:56:20.371812 392 external_mini_cluster.cc:1428] Setting key 04453a0db88e2a8c55c4c835a541d9ac
Jun 19 08:56:20 dist-test-slave-h4g5 krb5kdc[3409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:20.379798 3494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:20.370577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50349 (local address 127.0.98.62:35901)
0619 08:56:20.370637 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:20.370639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:20.370885 (+ 246us) server_negotiation.cc:407] Connection header received
0619 08:56:20.371377 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:20.371379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:20.371389 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:20.371458 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:20.372646 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:20.373150 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:20.374387 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:20.374577 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:20.374634 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:20.377034 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:20.377036 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:20.377036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:20.377044 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:20.378320 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:20.378800 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:20.378801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:20.378802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:20.378860 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:20.379131 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:20.379131 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:20.379131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:20.379248 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:20.379297 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:20.379651 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 08:56:20.379727 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 08:56:20.380020 3781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:20.370541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35901 (local address 127.0.98.2:50349)
0619 08:56:20.370704 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:20.370784 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:20.371262 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:20.371547 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:20.371552 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:20.372326 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:20.372566 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:20.372571 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:20.373252 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:20.373253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:20.374304 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:20.374307 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:20.374649 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:20.374650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:20.374788 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:20.375284 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:20.375294 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:20.376913 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:20.378407 (+ 1494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:20.378414 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:20.378417 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:20.378709 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:20.378929 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:20.378929 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:20.378930 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:20.379071 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:20.379308 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:20.379314 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:20.379489 (+ 175us) client_negotiation.cc:769] Sending connection context
0619 08:56:20.379612 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 08:56:20.379781 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
I20260619 08:56:20.380592 3779 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35901
I20260619 08:56:20.380695 3779 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:20.380919 3779 heartbeater.cc:511] Master 127.0.98.62:35901 requested a full tablet report, sending...
I20260619 08:56:20.381439 3454 ts_manager.cc:194] Registered new tserver with Master: 6f602abaeacf4a95bc118acc0b665fc5 (127.0.98.2:40227)
I20260619 08:56:20.381827 3454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50349
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 08:56:20.398530 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35901
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:20.477331 3784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:20.477556 3784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:20.477579 3784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:20.478782 3784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:20.478832 3784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:20.478859 3784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 08:56:20.480330 3784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:20.480542 3784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:20.480756 3784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:20.485569 3790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:20.485577 3789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:20.485823 3784 server_base.cc:1034] running on GCE node
W20260619 08:56:20.485577 3792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:20.486059 3784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:20.486304 3784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:20.487485 3784 hybrid_clock.cc:648] HybridClock initialized: now 1781859380487469 us; error 27 us; skew 500 ppm
Jun 19 08:56:20 dist-test-slave-h4g5 krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:20.490155 3784 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 08:56:20.490751 3784 webserver.cc:460] Webserver started at http://127.0.98.3:37635/ using document root <none> and password file <none>
I20260619 08:56:20.490998 3784 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:20.491047 3784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:20.491161 3784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:20.492252 3784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-2/data/instance:
uuid: "f06389e5e9c248d28314e596015a0866"
format_stamp: "Formatted at 2026-06-19 08:56:20 on dist-test-slave-h4g5"
server_key: "776c9b03301170550c575eb628f9a827"
server_key_iv: "ec62c687d4fe068bb51082fd20f7d77b"
server_key_version: "encryptionkey@0"
I20260619 08:56:20.492589 3784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f06389e5e9c248d28314e596015a0866"
format_stamp: "Formatted at 2026-06-19 08:56:20 on dist-test-slave-h4g5"
server_key: "776c9b03301170550c575eb628f9a827"
server_key_iv: "ec62c687d4fe068bb51082fd20f7d77b"
server_key_version: "encryptionkey@0"
I20260619 08:56:20.493845 3784 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:56:20.494613 3799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:20.494802 3784 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:20.494880 3784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-2/wal
uuid: "f06389e5e9c248d28314e596015a0866"
format_stamp: "Formatted at 2026-06-19 08:56:20 on dist-test-slave-h4g5"
server_key: "776c9b03301170550c575eb628f9a827"
server_key_iv: "ec62c687d4fe068bb51082fd20f7d77b"
server_key_version: "encryptionkey@0"
I20260619 08:56:20.494940 3784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:20.501384 3784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:20.503382 3784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:20.503515 3784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:20.503724 3784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:20.503999 3784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:20.504032 3784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:20.504058 3784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:20.504112 3784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:20.509455 3784 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44591
I20260619 08:56:20.509510 3912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44591 every 8 connection(s)
I20260619 08:56:20.509792 3784 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
I20260619 08:56:20.513146 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 3784
I20260619 08:56:20.513252 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859361540516-392-0/minicluster-data/ts-2/wal/instance
I20260619 08:56:20.513449 392 external_mini_cluster.cc:1428] Setting key 5d46b1291a3b5a7f267d749c02d3820d
Jun 19 08:56:20 dist-test-slave-h4g5 krb5kdc[3409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:20.518951 3494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:20.510476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51761 (local address 127.0.98.62:35901)
0619 08:56:20.510536 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:20.510538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:20.510743 (+ 205us) server_negotiation.cc:407] Connection header received
0619 08:56:20.511219 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:20.511220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:20.511231 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:20.511292 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:20.512275 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:20.512776 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:20.514076 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:20.514355 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:20.514419 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:20.516595 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:20.516596 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:20.516596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:20.516603 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:20.517898 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:20.518262 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:20.518262 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:20.518263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:20.518310 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:20.518485 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:20.518486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:20.518486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:20.518582 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:20.518623 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:20.518831 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 08:56:20.518886 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 08:56:20.519152 3915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:20.510443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35901 (local address 127.0.98.3:51761)
0619 08:56:20.510600 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:20.510636 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:20.511116 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:20.511362 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:20.511366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:20.511931 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:20.512180 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:20.512185 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:20.512844 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:20.512844 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:20.513983 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:20.513986 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:20.514437 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:20.514438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:20.514567 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:20.515035 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:20.515043 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:20.516510 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:20.517975 (+ 1465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:20.517981 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:20.517983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:20.518203 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:20.518355 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:20.518356 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:20.518356 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:20.518445 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:20.518621 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:20.518624 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:20.518738 (+ 114us) client_negotiation.cc:769] Sending connection context
0619 08:56:20.518814 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 08:56:20.518931 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
I20260619 08:56:20.519584 3913 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35901
I20260619 08:56:20.519677 3913 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:20.519841 3913 heartbeater.cc:511] Master 127.0.98.62:35901 requested a full tablet report, sending...
I20260619 08:56:20.520218 3454 ts_manager.cc:194] Registered new tserver with Master: f06389e5e9c248d28314e596015a0866 (127.0.98.3:44591)
I20260619 08:56:20.520555 3454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51761
I20260619 08:56:20.525252 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 08:56:20 dist-test-slave-h4g5 krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859380, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 08:56:20 dist-test-slave-h4g5 krb5kdc[3409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859380, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:20.546962 3494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:20.539971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57684 (local address 127.0.98.62:35901)
0619 08:56:20.540055 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:20.540057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:20.540196 (+ 139us) server_negotiation.cc:407] Connection header received
0619 08:56:20.540271 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:20.540272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:20.540282 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:20.540329 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:20.540888 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:20.541373 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:20.542101 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:20.542285 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:20.542341 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:20.544709 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:20.544711 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:20.544711 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:20.544719 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:20.545917 (+ 1198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:20.546234 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:20.546235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:20.546235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:20.546288 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:20.546450 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:20.546451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:20.546452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:20.546601 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:20.546696 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:20.546797 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 08:56:20.546872 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 08:56:20.547173 3454 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57684
I20260619 08:56:20.547567 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 3516
I20260619 08:56:20.551471 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 3650
I20260619 08:56:20.555398 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 3784
I20260619 08:56:20.559190 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 3422
2026-06-19T08:56:20Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1727 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:56:20 dist-test-slave-h4g5 krb5kdc[3925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:56:20 dist-test-slave-h4g5 krb5kdc[3925](info): set up 2 sockets
Jun 19 08:56:20 dist-test-slave-h4g5 krb5kdc[3925](info): commencing operation
krb5kdc: starting...
W20260619 08:56:21.519809 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 08:56:21 dist-test-slave-h4g5 krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:56:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:56:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 08:56:21.632939 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43897
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43897
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:21.710242 3938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:21.710443 3938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:21.710471 3938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:21.711603 3938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:56:21.711643 3938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:21.711661 3938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:21.711678 3938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:56:21.711692 3938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:56:21.713155 3938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:21.713395 3938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:21.713596 3938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:21.718127 3946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:21.718129 3943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:21.718163 3944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:21.718156 3938 server_base.cc:1034] running on GCE node
I20260619 08:56:21.718470 3938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:21.718708 3938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:21.719873 3938 hybrid_clock.cc:648] HybridClock initialized: now 1781859381719859 us; error 28 us; skew 500 ppm
Jun 19 08:56:21 dist-test-slave-h4g5 krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:21.722277 3938 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 08:56:21.722692 3938 webserver.cc:460] Webserver started at http://127.0.98.62:36927/ using document root <none> and password file <none>
I20260619 08:56:21.722905 3938 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:21.722949 3938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:21.723069 3938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:21.724175 3938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "7db75390696948b3bbef502eb7d21193"
format_stamp: "Formatted at 2026-06-19 08:56:21 on dist-test-slave-h4g5"
server_key: "83c77c11bbd2bed3e3bd7a02c66eb45c"
server_key_iv: "a11d347cfece4660572bc774a492c4db"
server_key_version: "encryptionkey@0"
I20260619 08:56:21.724498 3938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "7db75390696948b3bbef502eb7d21193"
format_stamp: "Formatted at 2026-06-19 08:56:21 on dist-test-slave-h4g5"
server_key: "83c77c11bbd2bed3e3bd7a02c66eb45c"
server_key_iv: "a11d347cfece4660572bc774a492c4db"
server_key_version: "encryptionkey@0"
I20260619 08:56:21.725791 3938 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:21.726454 3953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:21.726608 3938 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:21.726691 3938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "7db75390696948b3bbef502eb7d21193"
format_stamp: "Formatted at 2026-06-19 08:56:21 on dist-test-slave-h4g5"
server_key: "83c77c11bbd2bed3e3bd7a02c66eb45c"
server_key_iv: "a11d347cfece4660572bc774a492c4db"
server_key_version: "encryptionkey@0"
I20260619 08:56:21.726745 3938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:21.751338 3938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:21.753508 3938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:21.753620 3938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:21.757277 3938 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43897
I20260619 08:56:21.757336 4005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43897 every 8 connection(s)
I20260619 08:56:21.757663 3938 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:56:21.758502 4006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:21.760774 4006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7db75390696948b3bbef502eb7d21193: Bootstrap starting.
I20260619 08:56:21.761303 4006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7db75390696948b3bbef502eb7d21193: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:21.761534 4006 log.cc:824] T 00000000000000000000000000000000 P 7db75390696948b3bbef502eb7d21193: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:21.762144 4006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7db75390696948b3bbef502eb7d21193: No bootstrap required, opened a new log
I20260619 08:56:21.763700 4006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7db75390696948b3bbef502eb7d21193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db75390696948b3bbef502eb7d21193" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43897 } }
I20260619 08:56:21.763841 4006 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7db75390696948b3bbef502eb7d21193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:21.763863 4006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7db75390696948b3bbef502eb7d21193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7db75390696948b3bbef502eb7d21193, State: Initialized, Role: FOLLOWER
I20260619 08:56:21.763974 4006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7db75390696948b3bbef502eb7d21193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db75390696948b3bbef502eb7d21193" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43897 } }
I20260619 08:56:21.764047 4006 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7db75390696948b3bbef502eb7d21193 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:56:21.764109 4006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7db75390696948b3bbef502eb7d21193 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:56:21.764154 4006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7db75390696948b3bbef502eb7d21193 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:21.764636 4006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7db75390696948b3bbef502eb7d21193 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db75390696948b3bbef502eb7d21193" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43897 } }
I20260619 08:56:21.764719 4006 leader_election.cc:302] T 00000000000000000000000000000000 P 7db75390696948b3bbef502eb7d21193 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7db75390696948b3bbef502eb7d21193; no voters:
I20260619 08:56:21.764853 4006 leader_election.cc:288] T 00000000000000000000000000000000 P 7db75390696948b3bbef502eb7d21193 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:56:21.764905 4009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7db75390696948b3bbef502eb7d21193 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:21.765064 4006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7db75390696948b3bbef502eb7d21193 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:56:21.765252 4009 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7db75390696948b3bbef502eb7d21193 [term 1 LEADER]: Becoming Leader. State: Replica: 7db75390696948b3bbef502eb7d21193, State: Running, Role: LEADER
I20260619 08:56:21.765342 4009 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7db75390696948b3bbef502eb7d21193 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db75390696948b3bbef502eb7d21193" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43897 } }
I20260619 08:56:21.765707 4010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7db75390696948b3bbef502eb7d21193 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7db75390696948b3bbef502eb7d21193" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db75390696948b3bbef502eb7d21193" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43897 } } }
I20260619 08:56:21.765806 4010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7db75390696948b3bbef502eb7d21193 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:21.765964 4011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7db75390696948b3bbef502eb7d21193 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7db75390696948b3bbef502eb7d21193. Latest consensus state: current_term: 1 leader_uuid: "7db75390696948b3bbef502eb7d21193" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7db75390696948b3bbef502eb7d21193" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43897 } } }
I20260619 08:56:21.766007 4018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:56:21.766023 4011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7db75390696948b3bbef502eb7d21193 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:21.766583 4018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:56:21.767575 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 3938
I20260619 08:56:21.767675 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 08:56:21.767836 392 external_mini_cluster.cc:1428] Setting key a9ed563b91f894f9c9975028ec449e76
I20260619 08:56:21.768242 4018 catalog_manager.cc:1347] Generated new cluster ID: d34a381b9d3846fcbcc77acd7a920611
I20260619 08:56:21.768296 4018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 08:56:21 dist-test-slave-h4g5 krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:21.777887 4018 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:56:21.778421 4018 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:56:21.782135 4029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:21.772278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57634 (local address 127.0.98.62:43897)
0619 08:56:21.772464 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:21.772474 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:21.772505 (+ 31us) server_negotiation.cc:407] Connection header received
0619 08:56:21.772610 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:21.772616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:21.772663 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:21.772809 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:21.773536 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:21.774497 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:21.775268 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:21.775745 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:21.775863 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:21.778413 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:21.778419 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:21.778419 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:21.778432 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:21.780479 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:21.780904 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:21.780907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:21.780908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:21.781018 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:21.781216 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:21.781217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:21.781217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:21.781495 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:21.781639 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:21.781735 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 08:56:21.781886 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":90,"threads_started":1}
I20260619 08:56:21.795691 4018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7db75390696948b3bbef502eb7d21193: Generated new TSK 0
I20260619 08:56:21.795938 4018 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 08:56:21.820062 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43897
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:21.898092 4032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:21.898298 4032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:21.898319 4032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:21.899420 4032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:21.899453 4032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:21.899477 4032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 08:56:21.900987 4032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:21.901175 4032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:21.901348 4032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:21.906077 4037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:21.906080 4040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:21.906327 4032 server_base.cc:1034] running on GCE node
W20260619 08:56:21.906077 4038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:21.906587 4032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:21.906821 4032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:21.907994 4032 hybrid_clock.cc:648] HybridClock initialized: now 1781859381907979 us; error 27 us; skew 500 ppm
Jun 19 08:56:21 dist-test-slave-h4g5 krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:21.910374 4032 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 08:56:21.910785 4032 webserver.cc:460] Webserver started at http://127.0.98.1:39395/ using document root <none> and password file <none>
I20260619 08:56:21.910985 4032 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:21.911024 4032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:21.911116 4032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:21.912132 4032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-0/data/instance:
uuid: "0d81a3d343334d6dac396c0c57ff71ef"
format_stamp: "Formatted at 2026-06-19 08:56:21 on dist-test-slave-h4g5"
server_key: "d58c027b853e8e5b3ed477ff21ea7796"
server_key_iv: "1649b30093bcf50222fccb6060a8a93d"
server_key_version: "encryptionkey@0"
I20260619 08:56:21.912434 4032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0d81a3d343334d6dac396c0c57ff71ef"
format_stamp: "Formatted at 2026-06-19 08:56:21 on dist-test-slave-h4g5"
server_key: "d58c027b853e8e5b3ed477ff21ea7796"
server_key_iv: "1649b30093bcf50222fccb6060a8a93d"
server_key_version: "encryptionkey@0"
I20260619 08:56:21.913604 4032 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:56:21.914284 4047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:21.914440 4032 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 08:56:21.914521 4032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-0/wal
uuid: "0d81a3d343334d6dac396c0c57ff71ef"
format_stamp: "Formatted at 2026-06-19 08:56:21 on dist-test-slave-h4g5"
server_key: "d58c027b853e8e5b3ed477ff21ea7796"
server_key_iv: "1649b30093bcf50222fccb6060a8a93d"
server_key_version: "encryptionkey@0"
I20260619 08:56:21.914578 4032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:21.924772 4032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:21.926645 4032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:21.926769 4032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:21.926959 4032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:21.927219 4032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:21.927248 4032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:21.927269 4032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:21.927280 4032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:21.932760 4032 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37009
I20260619 08:56:21.932828 4160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37009 every 8 connection(s)
I20260619 08:56:21.933122 4032 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
I20260619 08:56:21.934432 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 4032
I20260619 08:56:21.934540 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-0/wal/instance
I20260619 08:56:21.934777 392 external_mini_cluster.cc:1428] Setting key ffa62851af14a47114fe5dd50bc05dbc
Jun 19 08:56:21 dist-test-slave-h4g5 krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 08:56:21.943022 4029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:21.933840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52693 (local address 127.0.98.62:43897)
0619 08:56:21.933906 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:21.933909 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:21.934171 (+ 262us) server_negotiation.cc:407] Connection header received
0619 08:56:21.935386 (+ 1215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:21.935387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:21.935396 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:21.935445 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:21.936319 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:21.936885 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:21.938101 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:21.938345 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:21.938405 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:21.940550 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:21.940552 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:21.940552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:21.940560 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:21.941806 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:21.942175 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:21.942176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:21.942176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:21.942227 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:21.942435 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:21.942436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:21.942436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:21.942539 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:21.942592 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:21.942850 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 08:56:21.942932 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 08:56:21.943197 4163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:21.933853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43897 (local address 127.0.98.1:52693)
0619 08:56:21.934049 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:21.934087 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:21.935289 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:21.935523 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:21.935528 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:21.935882 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:21.936203 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:21.936209 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:21.936988 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:21.936989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:21.937953 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:21.937955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:21.938420 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:21.938421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:21.938552 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:21.939107 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:21.939116 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:21.940464 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:21.941878 (+ 1414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:21.941884 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:21.941886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:21.942110 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:21.942280 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:21.942281 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:21.942281 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:21.942382 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:21.942592 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:21.942598 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:21.942733 (+ 135us) client_negotiation.cc:769] Sending connection context
0619 08:56:21.942830 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 08:56:21.942977 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":93,"threads_started":1}
I20260619 08:56:21.943668 4161 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43897
I20260619 08:56:21.943826 4161 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:21.943994 4161 heartbeater.cc:511] Master 127.0.98.62:43897 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 08:56:21.944571 3970 ts_manager.cc:194] Registered new tserver with Master: 0d81a3d343334d6dac396c0c57ff71ef (127.0.98.1:37009)
I20260619 08:56:21.945395 3970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52693
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 08:56:21.958612 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43897
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:22.038921 4166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:22.039134 4166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:22.039157 4166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:22.040333 4166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:22.040377 4166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:22.040405 4166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 08:56:22.041885 4166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:22.042212 4166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:22.042438 4166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:22.047116 4174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:22.047181 4171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:22.047098 4172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:22.047361 4166 server_base.cc:1034] running on GCE node
I20260619 08:56:22.047605 4166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:22.047827 4166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:22.048964 4166 hybrid_clock.cc:648] HybridClock initialized: now 1781859382048949 us; error 25 us; skew 500 ppm
Jun 19 08:56:22 dist-test-slave-h4g5 krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:22.054738 4166 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 08:56:22.055197 4166 webserver.cc:460] Webserver started at http://127.0.98.2:38707/ using document root <none> and password file <none>
I20260619 08:56:22.055394 4166 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:22.055441 4166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:22.055562 4166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:22.056607 4166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-1/data/instance:
uuid: "8b5f4d9f9d514450b6de4751e8d5f76c"
format_stamp: "Formatted at 2026-06-19 08:56:22 on dist-test-slave-h4g5"
server_key: "35614047531af6ccc2f51b68182a6d9c"
server_key_iv: "1077dabbe320e13e5eb9c751cb1d38c4"
server_key_version: "encryptionkey@0"
I20260619 08:56:22.056926 4166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8b5f4d9f9d514450b6de4751e8d5f76c"
format_stamp: "Formatted at 2026-06-19 08:56:22 on dist-test-slave-h4g5"
server_key: "35614047531af6ccc2f51b68182a6d9c"
server_key_iv: "1077dabbe320e13e5eb9c751cb1d38c4"
server_key_version: "encryptionkey@0"
I20260619 08:56:22.058090 4166 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:56:22.058805 4181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:22.058976 4166 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:22.059056 4166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-1/wal
uuid: "8b5f4d9f9d514450b6de4751e8d5f76c"
format_stamp: "Formatted at 2026-06-19 08:56:22 on dist-test-slave-h4g5"
server_key: "35614047531af6ccc2f51b68182a6d9c"
server_key_iv: "1077dabbe320e13e5eb9c751cb1d38c4"
server_key_version: "encryptionkey@0"
I20260619 08:56:22.059111 4166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:22.077003 4166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:22.078927 4166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:22.079066 4166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:22.079273 4166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:22.079702 4166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:22.079758 4166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:22.079792 4166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:22.079808 4166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:22.085292 4166 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38129
I20260619 08:56:22.085348 4294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38129 every 8 connection(s)
I20260619 08:56:22.085639 4166 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 08:56:22 dist-test-slave-h4g5 krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:22.093300 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 4166
I20260619 08:56:22.093396 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-1/wal/instance
I20260619 08:56:22.093567 392 external_mini_cluster.cc:1428] Setting key 1f4b6a6d7930dce6e8df3142320047b6
I20260619 08:56:22.094991 4029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:22.086314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56191 (local address 127.0.98.62:43897)
0619 08:56:22.086357 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:22.086359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:22.086544 (+ 185us) server_negotiation.cc:407] Connection header received
0619 08:56:22.087463 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:22.087464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:22.087476 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:22.087542 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:22.088136 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:22.088679 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:22.089744 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:22.089938 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:22.090001 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:22.092070 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:22.092072 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:22.092072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:22.092104 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:22.093653 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:22.094016 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:22.094017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:22.094018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:22.094083 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:22.094374 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:22.094375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:22.094375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:22.094491 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:22.094555 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:22.094851 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 08:56:22.094928 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 08:56:22.095197 4297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:22.086283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43897 (local address 127.0.98.2:56191)
0619 08:56:22.086425 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:22.086461 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:22.087340 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:22.087598 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:22.087601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:22.087851 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:22.088040 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:22.088045 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:22.088825 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:22.088826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:22.089623 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:22.089625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:22.090034 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:22.090035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:22.090154 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:22.090600 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:22.090607 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:22.091980 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:22.093738 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:22.093744 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:22.093746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:22.093953 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:22.094157 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:22.094158 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:22.094158 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:22.094288 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:22.094558 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:22.094564 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:22.094726 (+ 162us) client_negotiation.cc:769] Sending connection context
0619 08:56:22.094831 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 08:56:22.094986 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":82,"threads_started":1}
I20260619 08:56:22.095717 4295 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43897
I20260619 08:56:22.095804 4295 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:22.095968 4295 heartbeater.cc:511] Master 127.0.98.62:43897 requested a full tablet report, sending...
I20260619 08:56:22.096324 3970 ts_manager.cc:194] Registered new tserver with Master: 8b5f4d9f9d514450b6de4751e8d5f76c (127.0.98.2:38129)
I20260619 08:56:22.096789 3970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56191
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 08:56:22.118731 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43897
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:22.197757 4300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:22.197959 4300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:22.197989 4300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:22.199116 4300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:22.199152 4300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:22.199186 4300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 08:56:22.200770 4300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:22.200990 4300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:22.201185 4300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:22.205724 4306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:22.205727 4308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:22.205737 4305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:22.205998 4300 server_base.cc:1034] running on GCE node
I20260619 08:56:22.206184 4300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:22.206393 4300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:22.207564 4300 hybrid_clock.cc:648] HybridClock initialized: now 1781859382207548 us; error 27 us; skew 500 ppm
Jun 19 08:56:22 dist-test-slave-h4g5 krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:22.210028 4300 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 08:56:22.210448 4300 webserver.cc:460] Webserver started at http://127.0.98.3:38839/ using document root <none> and password file <none>
I20260619 08:56:22.210652 4300 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:22.210705 4300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:22.210834 4300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:22.211858 4300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-2/data/instance:
uuid: "768efbc40b2b434e89045c3879b3766b"
format_stamp: "Formatted at 2026-06-19 08:56:22 on dist-test-slave-h4g5"
server_key: "729ba3af9b59da32b08722327b56aafb"
server_key_iv: "3cd49c7f328fb6680ecc8322dd641402"
server_key_version: "encryptionkey@0"
I20260619 08:56:22.212229 4300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-2/wal/instance:
uuid: "768efbc40b2b434e89045c3879b3766b"
format_stamp: "Formatted at 2026-06-19 08:56:22 on dist-test-slave-h4g5"
server_key: "729ba3af9b59da32b08722327b56aafb"
server_key_iv: "3cd49c7f328fb6680ecc8322dd641402"
server_key_version: "encryptionkey@0"
I20260619 08:56:22.213476 4300 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:22.214156 4315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:22.214299 4300 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:22.214382 4300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-2/wal
uuid: "768efbc40b2b434e89045c3879b3766b"
format_stamp: "Formatted at 2026-06-19 08:56:22 on dist-test-slave-h4g5"
server_key: "729ba3af9b59da32b08722327b56aafb"
server_key_iv: "3cd49c7f328fb6680ecc8322dd641402"
server_key_version: "encryptionkey@0"
I20260619 08:56:22.214444 4300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:22.234980 4300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:22.237087 4300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:22.237222 4300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:22.237452 4300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:22.237733 4300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:22.237768 4300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:22.237797 4300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:22.237818 4300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:22.243268 4300 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37899
I20260619 08:56:22.243361 4428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37899 every 8 connection(s)
I20260619 08:56:22.243619 4300 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 08:56:22 dist-test-slave-h4g5 krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:22.253175 4029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:22.244359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46801 (local address 127.0.98.62:43897)
0619 08:56:22.244403 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:22.244405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:22.244671 (+ 266us) server_negotiation.cc:407] Connection header received
0619 08:56:22.245591 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:22.245592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:22.245605 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:22.245662 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:22.246289 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:22.246791 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:22.247829 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:22.248035 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:22.248158 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:22.250364 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:22.250366 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:22.250367 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:22.250374 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:22.251656 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:22.252114 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:22.252115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:22.252116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:22.252181 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:22.252491 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:22.252492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:22.252493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:22.252634 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:22.252709 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:22.253019 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 08:56:22.253101 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 08:56:22.253337 4431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:22.244333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43897 (local address 127.0.98.3:46801)
0619 08:56:22.244513 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:22.244558 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:22.245476 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:22.245729 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:22.245732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:22.246000 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:22.246191 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:22.246196 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:22.246916 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:22.246917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:22.247728 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:22.247730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:22.248161 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:22.248162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:22.248278 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:22.248772 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:22.248782 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:22.250239 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:22.251750 (+ 1511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:22.251757 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:22.251759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:22.251997 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:22.252264 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:22.252265 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:22.252265 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:22.252398 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:22.252713 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:22.252718 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:22.252848 (+ 130us) client_negotiation.cc:769] Sending connection context
0619 08:56:22.252945 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 08:56:22.253082 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":73,"threads_started":1}
I20260619 08:56:22.253384 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 4300
I20260619 08:56:22.253473 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859361540516-392-0/minicluster-data/ts-2/wal/instance
I20260619 08:56:22.253644 392 external_mini_cluster.cc:1428] Setting key 58b18985b173f0189aad0818517c80d1
I20260619 08:56:22.254166 4429 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43897
I20260619 08:56:22.254292 4429 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:22.254525 4429 heartbeater.cc:511] Master 127.0.98.62:43897 requested a full tablet report, sending...
I20260619 08:56:22.254858 3969 ts_manager.cc:194] Registered new tserver with Master: 768efbc40b2b434e89045c3879b3766b (127.0.98.3:37899)
I20260619 08:56:22.255244 3969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46801
I20260619 08:56:22.265571 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 08:56:22 dist-test-slave-h4g5 krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 08:56:22 dist-test-slave-h4g5 krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 08:56:22.288236 4436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:22.280768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38178 (local address 127.0.98.1:37009)
0619 08:56:22.280950 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:22.280955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:22.280965 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:56:22.280991 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:22.280993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:22.281006 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:22.281073 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:22.281610 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:22.282335 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:22.283099 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:22.283429 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:22.283486 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:22.285664 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:22.285671 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:22.285671 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:22.285677 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:22.287094 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:22.287423 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:22.287425 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:22.287425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:22.287485 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:22.287686 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:22.287687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:22.287687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:22.287840 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:22.287944 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:22.288053 (+ 109us) server_negotiation.cc:299] Negotiation successful
0619 08:56:22.288145 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":99,"threads_started":1}
Jun 19 08:56:22 dist-test-slave-h4g5 krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:22.295589 4029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:22.289376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57638 (local address 127.0.98.62:43897)
0619 08:56:22.289427 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:22.289429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:22.289437 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:56:22.289502 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:22.289503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:22.289512 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:22.289557 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:22.289997 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:22.290468 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:22.291113 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:22.291310 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:22.291363 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:22.293396 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:22.293398 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:22.293398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:22.293405 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:22.294578 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:22.294900 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:22.294901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:22.294902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:22.294951 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:22.295104 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:22.295105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:22.295105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:22.295239 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:22.295332 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:22.295421 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 08:56:22.295496 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260619 08:56:22.295846 3969 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57638
I20260619 08:56:22.296182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 4032
I20260619 08:56:22.300604 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 4166
I20260619 08:56:22.304415 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 4300
I20260619 08:56:22.307994 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 3938
2026-06-19T08:56:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1748 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:56:22 dist-test-slave-h4g5 krb5kdc[4441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:56:22 dist-test-slave-h4g5 krb5kdc[4441](info): set up 2 sockets
Jun 19 08:56:22 dist-test-slave-h4g5 krb5kdc[4441](info): commencing operation
krb5kdc: starting...
W20260619 08:56:23.255797 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 08:56:23 dist-test-slave-h4g5 krb5kdc[4441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:56:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:56:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 08:56:23.367048 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46731
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:23.444654 4454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:23.444861 4454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:23.444893 4454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:23.446019 4454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:56:23.446059 4454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:23.446079 4454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:23.446095 4454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:56:23.446111 4454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:56:23.447582 4454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:23.447829 4454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:23.448042 4454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:23.452661 4459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:23.452666 4462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:23.452688 4460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:23.453198 4454 server_base.cc:1034] running on GCE node
I20260619 08:56:23.453408 4454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:23.453648 4454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:23.454784 4454 hybrid_clock.cc:648] HybridClock initialized: now 1781859383454770 us; error 28 us; skew 500 ppm
Jun 19 08:56:23 dist-test-slave-h4g5 krb5kdc[4441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:23.457224 4454 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 08:56:23.457531 4454 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:23.457581 4454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:23.457701 4454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:23.458715 4454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "f98c612779404e67acfb716dea96fbb3"
format_stamp: "Formatted at 2026-06-19 08:56:23 on dist-test-slave-h4g5"
server_key: "a2325219e4a7246283119ca00b654902"
server_key_iv: "5166b847a8c62de00682771d092e0a27"
server_key_version: "encryptionkey@0"
I20260619 08:56:23.459036 4454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "f98c612779404e67acfb716dea96fbb3"
format_stamp: "Formatted at 2026-06-19 08:56:23 on dist-test-slave-h4g5"
server_key: "a2325219e4a7246283119ca00b654902"
server_key_iv: "5166b847a8c62de00682771d092e0a27"
server_key_version: "encryptionkey@0"
I20260619 08:56:23.460208 4454 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:56:23.460907 4468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:23.461090 4454 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:23.461171 4454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "f98c612779404e67acfb716dea96fbb3"
format_stamp: "Formatted at 2026-06-19 08:56:23 on dist-test-slave-h4g5"
server_key: "a2325219e4a7246283119ca00b654902"
server_key_iv: "5166b847a8c62de00682771d092e0a27"
server_key_version: "encryptionkey@0"
I20260619 08:56:23.461230 4454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:23.497857 4454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:23.500005 4454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:23.500176 4454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:23.503808 4454 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46731
I20260619 08:56:23.503841 4520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46731 every 8 connection(s)
I20260619 08:56:23.504114 4454 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:56:23.504956 4521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:23.507123 4521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f98c612779404e67acfb716dea96fbb3: Bootstrap starting.
I20260619 08:56:23.507638 4521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f98c612779404e67acfb716dea96fbb3: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:23.507860 4521 log.cc:824] T 00000000000000000000000000000000 P f98c612779404e67acfb716dea96fbb3: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:23.508458 4521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f98c612779404e67acfb716dea96fbb3: No bootstrap required, opened a new log
I20260619 08:56:23.509967 4521 raft_consensus.cc:348] T 00000000000000000000000000000000 P f98c612779404e67acfb716dea96fbb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f98c612779404e67acfb716dea96fbb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46731 } }
I20260619 08:56:23.510102 4521 raft_consensus.cc:374] T 00000000000000000000000000000000 P f98c612779404e67acfb716dea96fbb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:23.510123 4521 raft_consensus.cc:729] T 00000000000000000000000000000000 P f98c612779404e67acfb716dea96fbb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f98c612779404e67acfb716dea96fbb3, State: Initialized, Role: FOLLOWER
I20260619 08:56:23.510223 4521 consensus_queue.cc:260] T 00000000000000000000000000000000 P f98c612779404e67acfb716dea96fbb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f98c612779404e67acfb716dea96fbb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46731 } }
I20260619 08:56:23.510274 4521 raft_consensus.cc:388] T 00000000000000000000000000000000 P f98c612779404e67acfb716dea96fbb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:56:23.510300 4521 raft_consensus.cc:482] T 00000000000000000000000000000000 P f98c612779404e67acfb716dea96fbb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:56:23.510339 4521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f98c612779404e67acfb716dea96fbb3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:23.511076 4521 raft_consensus.cc:504] T 00000000000000000000000000000000 P f98c612779404e67acfb716dea96fbb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f98c612779404e67acfb716dea96fbb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46731 } }
I20260619 08:56:23.511168 4521 leader_election.cc:302] T 00000000000000000000000000000000 P f98c612779404e67acfb716dea96fbb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f98c612779404e67acfb716dea96fbb3; no voters:
I20260619 08:56:23.511303 4521 leader_election.cc:288] T 00000000000000000000000000000000 P f98c612779404e67acfb716dea96fbb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:56:23.511351 4524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f98c612779404e67acfb716dea96fbb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:23.511533 4524 raft_consensus.cc:686] T 00000000000000000000000000000000 P f98c612779404e67acfb716dea96fbb3 [term 1 LEADER]: Becoming Leader. State: Replica: f98c612779404e67acfb716dea96fbb3, State: Running, Role: LEADER
I20260619 08:56:23.511543 4521 sys_catalog.cc:564] T 00000000000000000000000000000000 P f98c612779404e67acfb716dea96fbb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:56:23.511678 4524 consensus_queue.cc:237] T 00000000000000000000000000000000 P f98c612779404e67acfb716dea96fbb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f98c612779404e67acfb716dea96fbb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46731 } }
I20260619 08:56:23.511883 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 4454
I20260619 08:56:23.511977 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 08:56:23.512218 392 external_mini_cluster.cc:1428] Setting key 88187833ce8d0e48a93bb68a214f6328
I20260619 08:56:23.512207 4525 sys_catalog.cc:455] T 00000000000000000000000000000000 P f98c612779404e67acfb716dea96fbb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f98c612779404e67acfb716dea96fbb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f98c612779404e67acfb716dea96fbb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46731 } } }
I20260619 08:56:23.512271 4525 sys_catalog.cc:458] T 00000000000000000000000000000000 P f98c612779404e67acfb716dea96fbb3 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:23.512562 4526 sys_catalog.cc:455] T 00000000000000000000000000000000 P f98c612779404e67acfb716dea96fbb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f98c612779404e67acfb716dea96fbb3. Latest consensus state: current_term: 1 leader_uuid: "f98c612779404e67acfb716dea96fbb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f98c612779404e67acfb716dea96fbb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46731 } } }
I20260619 08:56:23.512625 4526 sys_catalog.cc:458] T 00000000000000000000000000000000 P f98c612779404e67acfb716dea96fbb3 [sys.catalog]: This master's current role is: LEADER
W20260619 08:56:23.513324 4541 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f98c612779404e67acfb716dea96fbb3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 08:56:23.513365 4541 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 08:56:23.513466 4533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:56:23.513648 4533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:56:23.515055 4533 catalog_manager.cc:1347] Generated new cluster ID: cb425a91bdd14ef09b38727522fc300a
I20260619 08:56:23.515105 4533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 08:56:23 dist-test-slave-h4g5 krb5kdc[4441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:23.520675 4533 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:56:23.521173 4533 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:56:23.526954 4533 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f98c612779404e67acfb716dea96fbb3: Generated new TSK 0
I20260619 08:56:23.527158 4533 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:56:23.527236 4542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:23.512835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43468 (local address 127.0.98.62:46731)
0619 08:56:23.513466 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:23.513475 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:23.513499 (+ 24us) server_negotiation.cc:407] Connection header received
0619 08:56:23.513612 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:23.513618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:23.513668 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:23.513798 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:23.514433 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:23.515792 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:23.516988 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:23.517621 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:23.517740 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:23.520350 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:23.520355 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:23.520355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:23.520366 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:23.525568 (+ 5202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:23.525971 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:23.525974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:23.525974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:23.526083 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:23.526260 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:23.526261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:23.526261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:23.526545 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:23.526703 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:23.526811 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 08:56:23.527011 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":66,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 08:56:23.550630 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46731
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:23.630136 4547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:23.630334 4547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:23.630357 4547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:23.631479 4547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:23.631515 4547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:23.631541 4547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 08:56:23.633669 4547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:23.633908 4547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:23.634166 4547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:23.638918 4555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:23.638988 4553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:23.638918 4552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:23.639099 4547 server_base.cc:1034] running on GCE node
I20260619 08:56:23.639333 4547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:23.639547 4547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:23.640679 4547 hybrid_clock.cc:648] HybridClock initialized: now 1781859383640666 us; error 26 us; skew 500 ppm
Jun 19 08:56:23 dist-test-slave-h4g5 krb5kdc[4441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:23.642961 4547 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 08:56:23.643241 4547 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:23.643288 4547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:23.643411 4547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:23.644490 4547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-0/data/instance:
uuid: "539ed6d90dd74f89981157697fba99cd"
format_stamp: "Formatted at 2026-06-19 08:56:23 on dist-test-slave-h4g5"
server_key: "8294b27d5efc8ecf56b4253959dce5e7"
server_key_iv: "337bf2a39e8aa86b7118c5cce2652f3b"
server_key_version: "encryptionkey@0"
I20260619 08:56:23.644819 4547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-0/wal/instance:
uuid: "539ed6d90dd74f89981157697fba99cd"
format_stamp: "Formatted at 2026-06-19 08:56:23 on dist-test-slave-h4g5"
server_key: "8294b27d5efc8ecf56b4253959dce5e7"
server_key_iv: "337bf2a39e8aa86b7118c5cce2652f3b"
server_key_version: "encryptionkey@0"
I20260619 08:56:23.645938 4547 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:56:23.646538 4561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:23.646685 4547 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:23.646761 4547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-0/wal
uuid: "539ed6d90dd74f89981157697fba99cd"
format_stamp: "Formatted at 2026-06-19 08:56:23 on dist-test-slave-h4g5"
server_key: "8294b27d5efc8ecf56b4253959dce5e7"
server_key_iv: "337bf2a39e8aa86b7118c5cce2652f3b"
server_key_version: "encryptionkey@0"
I20260619 08:56:23.646826 4547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:23.661331 4547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:23.663347 4547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:23.663491 4547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:23.663734 4547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:23.664038 4547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:23.664070 4547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:23.664124 4547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:23.664147 4547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:23.669482 4547 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35161
I20260619 08:56:23.669538 4674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35161 every 8 connection(s)
I20260619 08:56:23.669755 4547 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
I20260619 08:56:23.675042 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 4547
I20260619 08:56:23.675148 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-0/wal/instance
Jun 19 08:56:23 dist-test-slave-h4g5 krb5kdc[4441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:23.675333 392 external_mini_cluster.cc:1428] Setting key a8be985774d6a4e57c9e0f1373f6cfcd
I20260619 08:56:23.678287 4542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:23.670401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51337 (local address 127.0.98.62:46731)
0619 08:56:23.670456 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:23.670459 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:23.670651 (+ 192us) server_negotiation.cc:407] Connection header received
0619 08:56:23.671086 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:23.671087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:23.671098 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:23.671148 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:23.671997 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:23.672518 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:23.673447 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:23.673630 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:23.673674 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:23.675745 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:23.675747 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:23.675747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:23.675754 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:23.677039 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:23.677417 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:23.677418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:23.677418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:23.677477 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:23.677689 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:23.677689 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:23.677690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:23.677802 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:23.677856 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:23.678112 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 08:56:23.678200 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 08:56:23.678454 4677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:23.670385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46731 (local address 127.0.98.1:51337)
0619 08:56:23.670524 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:23.670573 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:23.671013 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:23.671201 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:23.671205 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:23.671448 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:23.671922 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:23.671927 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:23.672593 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:23.672593 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:23.673378 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:23.673380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:23.673684 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:23.673685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:23.673778 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:23.674252 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:23.674260 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:23.675667 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:23.677133 (+ 1466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:23.677140 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:23.677142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:23.677352 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:23.677527 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:23.677528 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:23.677528 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:23.677636 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:23.677854 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:23.677860 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:23.677994 (+ 134us) client_negotiation.cc:769] Sending connection context
0619 08:56:23.678081 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 08:56:23.678230 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":74,"threads_started":1}
I20260619 08:56:23.678869 4675 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46731
I20260619 08:56:23.678952 4675 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:23.679082 4675 heartbeater.cc:511] Master 127.0.98.62:46731 requested a full tablet report, sending...
I20260619 08:56:23.679590 4485 ts_manager.cc:194] Registered new tserver with Master: 539ed6d90dd74f89981157697fba99cd (127.0.98.1:35161)
I20260619 08:56:23.680527 4485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51337
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 08:56:23.700837 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46731
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:23.779340 4680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:23.779536 4680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:23.779567 4680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:23.780701 4680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:23.780738 4680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:23.780772 4680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 08:56:23.782212 4680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:23.782434 4680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:23.782644 4680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:23.787339 4685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:23.787359 4686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:23.787460 4688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:23.787364 4680 server_base.cc:1034] running on GCE node
I20260619 08:56:23.787714 4680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:23.787930 4680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:23.789083 4680 hybrid_clock.cc:648] HybridClock initialized: now 1781859383789053 us; error 39 us; skew 500 ppm
Jun 19 08:56:23 dist-test-slave-h4g5 krb5kdc[4441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:23.791541 4680 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 08:56:23.791869 4680 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:23.791920 4680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:23.792043 4680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:23.793064 4680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-1/data/instance:
uuid: "777d8ce7dd4d4bc08ef85971a69083da"
format_stamp: "Formatted at 2026-06-19 08:56:23 on dist-test-slave-h4g5"
server_key: "928f646fb726eb51d1af8064d211e610"
server_key_iv: "46024e5c94c255cb1839adf2d53aa83c"
server_key_version: "encryptionkey@0"
I20260619 08:56:23.793385 4680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-1/wal/instance:
uuid: "777d8ce7dd4d4bc08ef85971a69083da"
format_stamp: "Formatted at 2026-06-19 08:56:23 on dist-test-slave-h4g5"
server_key: "928f646fb726eb51d1af8064d211e610"
server_key_iv: "46024e5c94c255cb1839adf2d53aa83c"
server_key_version: "encryptionkey@0"
I20260619 08:56:23.794553 4680 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:56:23.795301 4694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:23.795496 4680 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:23.795580 4680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-1/wal
uuid: "777d8ce7dd4d4bc08ef85971a69083da"
format_stamp: "Formatted at 2026-06-19 08:56:23 on dist-test-slave-h4g5"
server_key: "928f646fb726eb51d1af8064d211e610"
server_key_iv: "46024e5c94c255cb1839adf2d53aa83c"
server_key_version: "encryptionkey@0"
I20260619 08:56:23.795643 4680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:23.809917 4680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:23.812177 4680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:23.812336 4680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:23.812553 4680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:23.812858 4680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:23.812893 4680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:23.812922 4680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:23.812939 4680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:23.818388 4680 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38627
I20260619 08:56:23.818430 4807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38627 every 8 connection(s)
I20260619 08:56:23.818681 4680 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 08:56:23 dist-test-slave-h4g5 krb5kdc[4441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:23.825402 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 4680
I20260619 08:56:23.825510 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-1/wal/instance
I20260619 08:56:23.825691 392 external_mini_cluster.cc:1428] Setting key b8a54e459d0cc17bfb85aa4ef83bcc3a
I20260619 08:56:23.827826 4542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:23.819336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34895 (local address 127.0.98.62:46731)
0619 08:56:23.819393 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:23.819395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:23.819591 (+ 196us) server_negotiation.cc:407] Connection header received
0619 08:56:23.820066 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:23.820068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:23.820112 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:23.820187 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:23.821104 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:23.821593 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:23.822623 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:23.822805 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:23.822857 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:23.824869 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:23.824870 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:23.824870 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:23.824877 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:23.826339 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:23.826768 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:23.826768 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:23.826768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:23.826820 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:23.827060 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:23.827060 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:23.827060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:23.827156 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:23.827199 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:23.827638 (+ 439us) server_negotiation.cc:299] Negotiation successful
0619 08:56:23.827726 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 08:56:23.827946 4810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:23.819312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46731 (local address 127.0.98.2:34895)
0619 08:56:23.819466 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:23.819507 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:23.819974 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:23.820245 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:23.820249 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:23.820524 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:23.821015 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:23.821020 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:23.821690 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:23.821691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:23.822528 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:23.822530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:23.822868 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:23.822869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:23.822963 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:23.823453 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:23.823462 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:23.824786 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:23.826396 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:23.826404 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:23.826406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:23.826699 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:23.826873 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:23.826874 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:23.826874 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:23.827002 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:23.827205 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:23.827210 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:23.827370 (+ 160us) client_negotiation.cc:769] Sending connection context
0619 08:56:23.827556 (+ 186us) client_negotiation.cc:240] Negotiation successful
0619 08:56:23.827751 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
I20260619 08:56:23.828461 4808 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46731
I20260619 08:56:23.828545 4808 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:23.828699 4808 heartbeater.cc:511] Master 127.0.98.62:46731 requested a full tablet report, sending...
I20260619 08:56:23.829082 4485 ts_manager.cc:194] Registered new tserver with Master: 777d8ce7dd4d4bc08ef85971a69083da (127.0.98.2:38627)
I20260619 08:56:23.829432 4485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34895
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 08:56:23.850193 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46731
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:23.929327 4813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:23.929528 4813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:23.929553 4813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:23.930689 4813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:23.930729 4813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:23.930757 4813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 08:56:23.932250 4813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:23.932463 4813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:23.932667 4813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:23.937305 4819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:23.937307 4818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:23.937364 4813 server_base.cc:1034] running on GCE node
W20260619 08:56:23.937314 4821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:23.937743 4813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:23.937963 4813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:23.939128 4813 hybrid_clock.cc:648] HybridClock initialized: now 1781859383939112 us; error 33 us; skew 500 ppm
Jun 19 08:56:23 dist-test-slave-h4g5 krb5kdc[4441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:23.941457 4813 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 08:56:23.941761 4813 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:23.941807 4813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:23.941893 4813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:23.942874 4813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-2/data/instance:
uuid: "3f4b671c6fa5424cbcd00ffbbe8a392d"
format_stamp: "Formatted at 2026-06-19 08:56:23 on dist-test-slave-h4g5"
server_key: "c8daed25f0abdb226ce5185b4a9f511c"
server_key_iv: "5922d123a4c060e8dd46bca928a8f136"
server_key_version: "encryptionkey@0"
I20260619 08:56:23.943157 4813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3f4b671c6fa5424cbcd00ffbbe8a392d"
format_stamp: "Formatted at 2026-06-19 08:56:23 on dist-test-slave-h4g5"
server_key: "c8daed25f0abdb226ce5185b4a9f511c"
server_key_iv: "5922d123a4c060e8dd46bca928a8f136"
server_key_version: "encryptionkey@0"
I20260619 08:56:23.944306 4813 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:56:23.944977 4827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:23.945142 4813 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:23.945219 4813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-2/wal
uuid: "3f4b671c6fa5424cbcd00ffbbe8a392d"
format_stamp: "Formatted at 2026-06-19 08:56:23 on dist-test-slave-h4g5"
server_key: "c8daed25f0abdb226ce5185b4a9f511c"
server_key_iv: "5922d123a4c060e8dd46bca928a8f136"
server_key_version: "encryptionkey@0"
I20260619 08:56:23.945286 4813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:23.952349 4813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:23.954303 4813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:23.954418 4813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:23.954617 4813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:23.954890 4813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:23.954921 4813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:23.954941 4813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:23.954963 4813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:23.960376 4813 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42263
I20260619 08:56:23.960417 4940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42263 every 8 connection(s)
I20260619 08:56:23.960669 4813 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
I20260619 08:56:23.964561 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 4813
I20260619 08:56:23.964685 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-2/wal/instance
I20260619 08:56:23.964864 392 external_mini_cluster.cc:1428] Setting key e2f0c70fda81f10846cf327160b57b36
Jun 19 08:56:23 dist-test-slave-h4g5 krb5kdc[4441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:23.969549 4542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:23.961389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46467 (local address 127.0.98.62:46731)
0619 08:56:23.961475 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:23.961477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:23.961639 (+ 162us) server_negotiation.cc:407] Connection header received
0619 08:56:23.962090 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:23.962091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:23.962101 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:23.962160 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:23.963070 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:23.963571 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:23.964755 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:23.964981 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:23.965035 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:23.967064 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:23.967066 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:23.967066 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:23.967073 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:23.968336 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:23.968700 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:23.968701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:23.968702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:23.968756 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:23.968963 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:23.968964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:23.968964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:23.969072 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:23.969126 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:23.969377 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 08:56:23.969462 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 08:56:23.969705 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:23.961319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46731 (local address 127.0.98.3:46467)
0619 08:56:23.961506 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:23.961547 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:23.961999 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:23.962224 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:23.962227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:23.962502 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:23.962976 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:23.962982 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:23.963652 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:23.963653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:23.964630 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:23.964632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:23.965062 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:23.965062 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:23.965187 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:23.965672 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:23.965680 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:23.966979 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:23.968408 (+ 1429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:23.968414 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:23.968416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:23.968622 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:23.968808 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:23.968809 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:23.968809 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:23.968909 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:23.969123 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:23.969129 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:23.969262 (+ 133us) client_negotiation.cc:769] Sending connection context
0619 08:56:23.969348 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 08:56:23.969498 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":79,"threads_started":1}
I20260619 08:56:23.970129 4941 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46731
I20260619 08:56:23.970216 4941 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:23.970348 4941 heartbeater.cc:511] Master 127.0.98.62:46731 requested a full tablet report, sending...
I20260619 08:56:23.970667 4485 ts_manager.cc:194] Registered new tserver with Master: 3f4b671c6fa5424cbcd00ffbbe8a392d (127.0.98.3:42263)
I20260619 08:56:23.971012 4485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46467
I20260619 08:56:23.976507 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:56:23.983242 4542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:23.977770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43480 (local address 127.0.98.62:46731)
0619 08:56:23.977827 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:23.977828 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:23.977933 (+ 105us) server_negotiation.cc:407] Connection header received
0619 08:56:23.977978 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:23.977978 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:23.977986 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:23.978035 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:23.978590 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:23.979056 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:23.979750 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:23.979920 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:23.979961 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:23.981124 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:23.981125 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:23.981125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:23.981131 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:23.982231 (+ 1100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:23.982546 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:23.982547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:23.982547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:23.982589 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:23.982723 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:23.982723 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:23.982723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:23.982854 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:23.982923 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:23.983117 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 08:56:23.983176 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 08:56:23.984997 4485 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:56:23.986503 4485 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:56:23.993105 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:23.989026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44326 (local address 127.0.98.2:38627)
0619 08:56:23.989179 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:23.989185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:23.989284 (+ 99us) server_negotiation.cc:407] Connection header received
0619 08:56:23.989350 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:23.989351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:23.989364 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:23.989442 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:23.989598 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:23.990484 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:23.992199 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:23.992689 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:23.992760 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:23.992823 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:23.992968 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 08:56:23.993012 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
I20260619 08:56:23.993113 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:23.988790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42263 (local address 127.0.0.1:40296)
0619 08:56:23.989129 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:23.989139 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:23.989184 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:23.989792 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:23.989793 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:23.989795 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:23.989828 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:23.989830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:23.991031 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:23.991032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:23.991885 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:23.991887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:23.992866 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:23.992867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:23.992918 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:23.992921 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:56:23.992960 (+ 39us) client_negotiation.cc:240] Negotiation successful
0619 08:56:23.993027 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":2,"thread_start_us":53,"threads_started":1}
I20260619 08:56:23.993129 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:23.988758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35161 (local address 127.0.0.1:51528)
0619 08:56:23.988883 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:23.988923 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:23.988991 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:23.989404 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:23.989407 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:23.989410 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:23.989487 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:23.989489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:23.990795 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:23.990796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:23.991859 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:23.991860 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:23.992810 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:23.992811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:23.992874 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:23.992879 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 08:56:23.992921 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 08:56:23.993023 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":31,"thread_start_us":77,"threads_started":1}
I20260619 08:56:23.993342 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:23.988799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38627 (local address 127.0.0.1:44326)
0619 08:56:23.989207 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:23.989219 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:23.989283 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:23.989449 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:23.989450 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:23.989452 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:23.989505 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:23.989507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:23.990584 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:23.990585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:23.991856 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:23.991859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:23.992774 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:23.992775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:23.992859 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:23.992875 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 08:56:23.992944 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 08:56:23.993291 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":158,"thread_start_us":79,"threads_started":1}
I20260619 08:56:23.993451 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:23.989007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40296 (local address 127.0.98.3:42263)
0619 08:56:23.989584 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:23.989588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:23.989596 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:56:23.989629 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:23.989631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:23.989646 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:23.989712 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:23.989907 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:23.990944 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:23.991990 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:23.992711 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:23.992781 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:23.992838 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:23.993326 (+ 488us) server_negotiation.cc:299] Negotiation successful
0619 08:56:23.993370 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":122,"threads_started":1}
I20260619 08:56:23.993454 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:23.988956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51528 (local address 127.0.98.1:35161)
0619 08:56:23.989242 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:23.989247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:23.989257 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:56:23.989298 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:23.989300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:23.989316 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:23.989387 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:23.989570 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:23.990695 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:23.991949 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:23.992711 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:23.993011 (+ 300us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:23.993072 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:23.993097 (+ 25us) server_negotiation.cc:299] Negotiation successful
0619 08:56:23.993136 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":86,"threads_started":1}
I20260619 08:56:23.994099 4609 tablet_service.cc:1467] Processing CreateTablet for tablet 799dc4b83e3f4a14b3ca241d056c527c (DEFAULT_TABLE table=test-table [id=bfea7d86a5ef48f198bfc78f127fc04a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:23.994119 4742 tablet_service.cc:1467] Processing CreateTablet for tablet 799dc4b83e3f4a14b3ca241d056c527c (DEFAULT_TABLE table=test-table [id=bfea7d86a5ef48f198bfc78f127fc04a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:23.994393 4609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 799dc4b83e3f4a14b3ca241d056c527c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:23.994292 4875 tablet_service.cc:1467] Processing CreateTablet for tablet 799dc4b83e3f4a14b3ca241d056c527c (DEFAULT_TABLE table=test-table [id=bfea7d86a5ef48f198bfc78f127fc04a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:23.994436 4742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 799dc4b83e3f4a14b3ca241d056c527c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:23.994568 4875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 799dc4b83e3f4a14b3ca241d056c527c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:23.997115 4955 tablet_bootstrap.cc:492] T 799dc4b83e3f4a14b3ca241d056c527c P 539ed6d90dd74f89981157697fba99cd: Bootstrap starting.
I20260619 08:56:23.997754 4957 tablet_bootstrap.cc:492] T 799dc4b83e3f4a14b3ca241d056c527c P 3f4b671c6fa5424cbcd00ffbbe8a392d: Bootstrap starting.
I20260619 08:56:23.997848 4955 tablet_bootstrap.cc:654] T 799dc4b83e3f4a14b3ca241d056c527c P 539ed6d90dd74f89981157697fba99cd: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:23.997900 4956 tablet_bootstrap.cc:492] T 799dc4b83e3f4a14b3ca241d056c527c P 777d8ce7dd4d4bc08ef85971a69083da: Bootstrap starting.
I20260619 08:56:23.998435 4957 tablet_bootstrap.cc:654] T 799dc4b83e3f4a14b3ca241d056c527c P 3f4b671c6fa5424cbcd00ffbbe8a392d: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:23.998461 4956 tablet_bootstrap.cc:654] T 799dc4b83e3f4a14b3ca241d056c527c P 777d8ce7dd4d4bc08ef85971a69083da: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:23.998466 4955 log.cc:824] T 799dc4b83e3f4a14b3ca241d056c527c P 539ed6d90dd74f89981157697fba99cd: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:23.998972 4957 log.cc:824] T 799dc4b83e3f4a14b3ca241d056c527c P 3f4b671c6fa5424cbcd00ffbbe8a392d: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:23.999049 4956 log.cc:824] T 799dc4b83e3f4a14b3ca241d056c527c P 777d8ce7dd4d4bc08ef85971a69083da: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:23.999573 4955 tablet_bootstrap.cc:492] T 799dc4b83e3f4a14b3ca241d056c527c P 539ed6d90dd74f89981157697fba99cd: No bootstrap required, opened a new log
I20260619 08:56:23.999629 4955 ts_tablet_manager.cc:1397] T 799dc4b83e3f4a14b3ca241d056c527c P 539ed6d90dd74f89981157697fba99cd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 08:56:23.999708 4957 tablet_bootstrap.cc:492] T 799dc4b83e3f4a14b3ca241d056c527c P 3f4b671c6fa5424cbcd00ffbbe8a392d: No bootstrap required, opened a new log
I20260619 08:56:23.999770 4957 ts_tablet_manager.cc:1397] T 799dc4b83e3f4a14b3ca241d056c527c P 3f4b671c6fa5424cbcd00ffbbe8a392d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:56:24.000468 4956 tablet_bootstrap.cc:492] T 799dc4b83e3f4a14b3ca241d056c527c P 777d8ce7dd4d4bc08ef85971a69083da: No bootstrap required, opened a new log
I20260619 08:56:24.000550 4956 ts_tablet_manager.cc:1397] T 799dc4b83e3f4a14b3ca241d056c527c P 777d8ce7dd4d4bc08ef85971a69083da: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 08:56:24.001307 4957 raft_consensus.cc:348] T 799dc4b83e3f4a14b3ca241d056c527c P 3f4b671c6fa5424cbcd00ffbbe8a392d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539ed6d90dd74f89981157697fba99cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35161 } } peers { permanent_uuid: "3f4b671c6fa5424cbcd00ffbbe8a392d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42263 } } peers { permanent_uuid: "777d8ce7dd4d4bc08ef85971a69083da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38627 } }
I20260619 08:56:24.001453 4957 raft_consensus.cc:374] T 799dc4b83e3f4a14b3ca241d056c527c P 3f4b671c6fa5424cbcd00ffbbe8a392d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:24.001484 4957 raft_consensus.cc:729] T 799dc4b83e3f4a14b3ca241d056c527c P 3f4b671c6fa5424cbcd00ffbbe8a392d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f4b671c6fa5424cbcd00ffbbe8a392d, State: Initialized, Role: FOLLOWER
I20260619 08:56:24.001609 4957 consensus_queue.cc:260] T 799dc4b83e3f4a14b3ca241d056c527c P 3f4b671c6fa5424cbcd00ffbbe8a392d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539ed6d90dd74f89981157697fba99cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35161 } } peers { permanent_uuid: "3f4b671c6fa5424cbcd00ffbbe8a392d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42263 } } peers { permanent_uuid: "777d8ce7dd4d4bc08ef85971a69083da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38627 } }
I20260619 08:56:24.001579 4955 raft_consensus.cc:348] T 799dc4b83e3f4a14b3ca241d056c527c P 539ed6d90dd74f89981157697fba99cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539ed6d90dd74f89981157697fba99cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35161 } } peers { permanent_uuid: "3f4b671c6fa5424cbcd00ffbbe8a392d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42263 } } peers { permanent_uuid: "777d8ce7dd4d4bc08ef85971a69083da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38627 } }
I20260619 08:56:24.001757 4955 raft_consensus.cc:374] T 799dc4b83e3f4a14b3ca241d056c527c P 539ed6d90dd74f89981157697fba99cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:24.001792 4955 raft_consensus.cc:729] T 799dc4b83e3f4a14b3ca241d056c527c P 539ed6d90dd74f89981157697fba99cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 539ed6d90dd74f89981157697fba99cd, State: Initialized, Role: FOLLOWER
I20260619 08:56:24.001919 4957 ts_tablet_manager.cc:1428] T 799dc4b83e3f4a14b3ca241d056c527c P 3f4b671c6fa5424cbcd00ffbbe8a392d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:56:24.001976 4941 heartbeater.cc:503] Master 127.0.98.62:46731 was elected leader, sending a full tablet report...
I20260619 08:56:24.001931 4955 consensus_queue.cc:260] T 799dc4b83e3f4a14b3ca241d056c527c P 539ed6d90dd74f89981157697fba99cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539ed6d90dd74f89981157697fba99cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35161 } } peers { permanent_uuid: "3f4b671c6fa5424cbcd00ffbbe8a392d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42263 } } peers { permanent_uuid: "777d8ce7dd4d4bc08ef85971a69083da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38627 } }
I20260619 08:56:24.002231 4955 ts_tablet_manager.cc:1428] T 799dc4b83e3f4a14b3ca241d056c527c P 539ed6d90dd74f89981157697fba99cd: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 08:56:24.002256 4675 heartbeater.cc:503] Master 127.0.98.62:46731 was elected leader, sending a full tablet report...
I20260619 08:56:24.002519 4956 raft_consensus.cc:348] T 799dc4b83e3f4a14b3ca241d056c527c P 777d8ce7dd4d4bc08ef85971a69083da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539ed6d90dd74f89981157697fba99cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35161 } } peers { permanent_uuid: "3f4b671c6fa5424cbcd00ffbbe8a392d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42263 } } peers { permanent_uuid: "777d8ce7dd4d4bc08ef85971a69083da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38627 } }
I20260619 08:56:24.002668 4956 raft_consensus.cc:374] T 799dc4b83e3f4a14b3ca241d056c527c P 777d8ce7dd4d4bc08ef85971a69083da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:24.002698 4956 raft_consensus.cc:729] T 799dc4b83e3f4a14b3ca241d056c527c P 777d8ce7dd4d4bc08ef85971a69083da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 777d8ce7dd4d4bc08ef85971a69083da, State: Initialized, Role: FOLLOWER
I20260619 08:56:24.002797 4956 consensus_queue.cc:260] T 799dc4b83e3f4a14b3ca241d056c527c P 777d8ce7dd4d4bc08ef85971a69083da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539ed6d90dd74f89981157697fba99cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35161 } } peers { permanent_uuid: "3f4b671c6fa5424cbcd00ffbbe8a392d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42263 } } peers { permanent_uuid: "777d8ce7dd4d4bc08ef85971a69083da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38627 } }
I20260619 08:56:24.003060 4956 ts_tablet_manager.cc:1428] T 799dc4b83e3f4a14b3ca241d056c527c P 777d8ce7dd4d4bc08ef85971a69083da: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 08:56:24.003085 4808 heartbeater.cc:503] Master 127.0.98.62:46731 was elected leader, sending a full tablet report...
W20260619 08:56:24.069193 4809 tablet.cc:2369] T 799dc4b83e3f4a14b3ca241d056c527c P 777d8ce7dd4d4bc08ef85971a69083da: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 08:56:24.170398 4676 tablet.cc:2369] T 799dc4b83e3f4a14b3ca241d056c527c P 539ed6d90dd74f89981157697fba99cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:56:24.174052 4962 raft_consensus.cc:482] T 799dc4b83e3f4a14b3ca241d056c527c P 539ed6d90dd74f89981157697fba99cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:56:24.174196 4962 raft_consensus.cc:504] T 799dc4b83e3f4a14b3ca241d056c527c P 539ed6d90dd74f89981157697fba99cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539ed6d90dd74f89981157697fba99cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35161 } } peers { permanent_uuid: "3f4b671c6fa5424cbcd00ffbbe8a392d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42263 } } peers { permanent_uuid: "777d8ce7dd4d4bc08ef85971a69083da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38627 } }
I20260619 08:56:24.174598 4962 leader_election.cc:288] T 799dc4b83e3f4a14b3ca241d056c527c P 539ed6d90dd74f89981157697fba99cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f4b671c6fa5424cbcd00ffbbe8a392d (127.0.98.3:42263), 777d8ce7dd4d4bc08ef85971a69083da (127.0.98.2:38627)
I20260619 08:56:24.177887 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:24.174927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59771 (local address 127.0.98.2:38627)
0619 08:56:24.174969 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:24.174971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:24.175113 (+ 142us) server_negotiation.cc:407] Connection header received
0619 08:56:24.175176 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:24.175177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:24.175188 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:24.175231 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:24.175372 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:24.175832 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:24.176974 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:24.177542 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:24.177585 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:24.177595 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:24.177771 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 08:56:24.177833 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 08:56:24.177910 4677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:24.174824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42263 (local address 127.0.98.1:56621)
0619 08:56:24.174920 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:24.174932 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:24.174992 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:24.175206 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:24.175206 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:24.175208 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:24.175278 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:24.175280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:24.175879 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:24.175880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:24.176833 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:24.176836 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:24.177590 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:24.177591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:24.177713 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:24.177716 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:56:24.177763 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 08:56:24.177807 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":31}
I20260619 08:56:24.177935 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:24.174878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38627 (local address 127.0.98.1:59771)
0619 08:56:24.175029 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:24.175038 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:24.175113 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:24.175245 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:24.175246 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:24.175247 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:24.175315 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:24.175316 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:24.175904 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:24.175905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:24.176877 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:24.176880 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:24.177643 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:24.177644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:24.177712 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:24.177717 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 08:56:24.177763 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 08:56:24.177807 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":6,"thread_start_us":107,"threads_started":1}
I20260619 08:56:24.178088 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:24.174932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56621 (local address 127.0.98.3:42263)
0619 08:56:24.174982 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:24.174984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:24.174992 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:56:24.175124 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:24.175125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:24.175134 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:24.175190 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:24.175338 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:24.175806 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:24.176949 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:24.177499 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:24.177585 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:24.177597 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:24.177979 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 08:56:24.178025 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 08:56:24.178265 4762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "799dc4b83e3f4a14b3ca241d056c527c" candidate_uuid: "539ed6d90dd74f89981157697fba99cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "777d8ce7dd4d4bc08ef85971a69083da" is_pre_election: true
I20260619 08:56:24.178443 4762 raft_consensus.cc:2454] T 799dc4b83e3f4a14b3ca241d056c527c P 777d8ce7dd4d4bc08ef85971a69083da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 539ed6d90dd74f89981157697fba99cd in term 0.
I20260619 08:56:24.178550 4895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "799dc4b83e3f4a14b3ca241d056c527c" candidate_uuid: "539ed6d90dd74f89981157697fba99cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f4b671c6fa5424cbcd00ffbbe8a392d" is_pre_election: true
I20260619 08:56:24.178656 4564 leader_election.cc:302] T 799dc4b83e3f4a14b3ca241d056c527c P 539ed6d90dd74f89981157697fba99cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 539ed6d90dd74f89981157697fba99cd, 777d8ce7dd4d4bc08ef85971a69083da; no voters:
I20260619 08:56:24.178686 4895 raft_consensus.cc:2454] T 799dc4b83e3f4a14b3ca241d056c527c P 3f4b671c6fa5424cbcd00ffbbe8a392d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 539ed6d90dd74f89981157697fba99cd in term 0.
I20260619 08:56:24.178783 4962 raft_consensus.cc:2781] T 799dc4b83e3f4a14b3ca241d056c527c P 539ed6d90dd74f89981157697fba99cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:56:24.178850 4962 raft_consensus.cc:482] T 799dc4b83e3f4a14b3ca241d056c527c P 539ed6d90dd74f89981157697fba99cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:56:24.178879 4962 raft_consensus.cc:3037] T 799dc4b83e3f4a14b3ca241d056c527c P 539ed6d90dd74f89981157697fba99cd [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:24.179527 4962 raft_consensus.cc:504] T 799dc4b83e3f4a14b3ca241d056c527c P 539ed6d90dd74f89981157697fba99cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539ed6d90dd74f89981157697fba99cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35161 } } peers { permanent_uuid: "3f4b671c6fa5424cbcd00ffbbe8a392d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42263 } } peers { permanent_uuid: "777d8ce7dd4d4bc08ef85971a69083da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38627 } }
I20260619 08:56:24.179653 4962 leader_election.cc:288] T 799dc4b83e3f4a14b3ca241d056c527c P 539ed6d90dd74f89981157697fba99cd [CANDIDATE]: Term 1 election: Requested vote from peers 3f4b671c6fa5424cbcd00ffbbe8a392d (127.0.98.3:42263), 777d8ce7dd4d4bc08ef85971a69083da (127.0.98.2:38627)
I20260619 08:56:24.179795 4762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "799dc4b83e3f4a14b3ca241d056c527c" candidate_uuid: "539ed6d90dd74f89981157697fba99cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "777d8ce7dd4d4bc08ef85971a69083da"
I20260619 08:56:24.179837 4895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "799dc4b83e3f4a14b3ca241d056c527c" candidate_uuid: "539ed6d90dd74f89981157697fba99cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f4b671c6fa5424cbcd00ffbbe8a392d"
I20260619 08:56:24.179867 4762 raft_consensus.cc:3037] T 799dc4b83e3f4a14b3ca241d056c527c P 777d8ce7dd4d4bc08ef85971a69083da [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:24.179900 4895 raft_consensus.cc:3037] T 799dc4b83e3f4a14b3ca241d056c527c P 3f4b671c6fa5424cbcd00ffbbe8a392d [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:24.180595 4895 raft_consensus.cc:2454] T 799dc4b83e3f4a14b3ca241d056c527c P 3f4b671c6fa5424cbcd00ffbbe8a392d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 539ed6d90dd74f89981157697fba99cd in term 1.
I20260619 08:56:24.180601 4762 raft_consensus.cc:2454] T 799dc4b83e3f4a14b3ca241d056c527c P 777d8ce7dd4d4bc08ef85971a69083da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 539ed6d90dd74f89981157697fba99cd in term 1.
I20260619 08:56:24.180732 4564 leader_election.cc:302] T 799dc4b83e3f4a14b3ca241d056c527c P 539ed6d90dd74f89981157697fba99cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 539ed6d90dd74f89981157697fba99cd, 777d8ce7dd4d4bc08ef85971a69083da; no voters:
I20260619 08:56:24.180840 4962 raft_consensus.cc:2781] T 799dc4b83e3f4a14b3ca241d056c527c P 539ed6d90dd74f89981157697fba99cd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:24.181025 4962 raft_consensus.cc:686] T 799dc4b83e3f4a14b3ca241d056c527c P 539ed6d90dd74f89981157697fba99cd [term 1 LEADER]: Becoming Leader. State: Replica: 539ed6d90dd74f89981157697fba99cd, State: Running, Role: LEADER
I20260619 08:56:24.181115 4962 consensus_queue.cc:237] T 799dc4b83e3f4a14b3ca241d056c527c P 539ed6d90dd74f89981157697fba99cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539ed6d90dd74f89981157697fba99cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35161 } } peers { permanent_uuid: "3f4b671c6fa5424cbcd00ffbbe8a392d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42263 } } peers { permanent_uuid: "777d8ce7dd4d4bc08ef85971a69083da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38627 } }
I20260619 08:56:24.181955 4484 catalog_manager.cc:5495] T 799dc4b83e3f4a14b3ca241d056c527c P 539ed6d90dd74f89981157697fba99cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 539ed6d90dd74f89981157697fba99cd (127.0.98.1). New cstate: current_term: 1 leader_uuid: "539ed6d90dd74f89981157697fba99cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "539ed6d90dd74f89981157697fba99cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f4b671c6fa5424cbcd00ffbbe8a392d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "777d8ce7dd4d4bc08ef85971a69083da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38627 } health_report { overall_health: UNKNOWN } } }
W20260619 08:56:24.211194 4942 tablet.cc:2369] T 799dc4b83e3f4a14b3ca241d056c527c P 3f4b671c6fa5424cbcd00ffbbe8a392d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:56:24.222929 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:24.220530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51530 (local address 127.0.98.1:35161)
0619 08:56:24.220581 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:24.220582 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:24.220591 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:56:24.220631 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:24.220632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:24.220652 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:24.220718 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:56:24.220861 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:24.221346 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:24.222144 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:24.222336 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:24.222389 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:24.222605 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:56:24.222711 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:56:24.222787 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:24.222832 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 08:56:24.222851 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 08:56:24.224965 4895 raft_consensus.cc:1261] T 799dc4b83e3f4a14b3ca241d056c527c P 3f4b671c6fa5424cbcd00ffbbe8a392d [term 1 FOLLOWER]: Refusing update from remote peer 539ed6d90dd74f89981157697fba99cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:56:24.224965 4762 raft_consensus.cc:1261] T 799dc4b83e3f4a14b3ca241d056c527c P 777d8ce7dd4d4bc08ef85971a69083da [term 1 FOLLOWER]: Refusing update from remote peer 539ed6d90dd74f89981157697fba99cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:56:24.225376 4965 consensus_queue.cc:1035] T 799dc4b83e3f4a14b3ca241d056c527c P 539ed6d90dd74f89981157697fba99cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "777d8ce7dd4d4bc08ef85971a69083da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:56:24.225538 4962 consensus_queue.cc:1035] T 799dc4b83e3f4a14b3ca241d056c527c P 539ed6d90dd74f89981157697fba99cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f4b671c6fa5424cbcd00ffbbe8a392d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:56:24.228823 4969 mvcc.cc:205] Tried to move back new op lower bound from 7298496037782261760 to 7298496037606486016. Current Snapshot: MvccSnapshot[applied={T|T < 7298496037782261760}]
I20260619 08:56:24.230466 4485 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:56:24.230556 4485 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:56:24.231293 4485 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f98c612779404e67acfb716dea96fbb3: Sending DeleteTablet for 3 replicas of tablet 799dc4b83e3f4a14b3ca241d056c527c
I20260619 08:56:24.231562 4609 tablet_service.cc:1514] Processing DeleteTablet for tablet 799dc4b83e3f4a14b3ca241d056c527c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51528
I20260619 08:56:24.231595 4742 tablet_service.cc:1514] Processing DeleteTablet for tablet 799dc4b83e3f4a14b3ca241d056c527c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44326
I20260619 08:56:24.231822 4875 tablet_service.cc:1514] Processing DeleteTablet for tablet 799dc4b83e3f4a14b3ca241d056c527c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40296
I20260619 08:56:24.231964 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 4547
I20260619 08:56:24.232569 4979 tablet_replica.cc:330] T 799dc4b83e3f4a14b3ca241d056c527c P 777d8ce7dd4d4bc08ef85971a69083da: stopping tablet replica
I20260619 08:56:24.232697 4980 tablet_replica.cc:330] T 799dc4b83e3f4a14b3ca241d056c527c P 3f4b671c6fa5424cbcd00ffbbe8a392d: stopping tablet replica
I20260619 08:56:24.232854 4980 raft_consensus.cc:2229] T 799dc4b83e3f4a14b3ca241d056c527c P 3f4b671c6fa5424cbcd00ffbbe8a392d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:56:24.232999 4979 raft_consensus.cc:2229] T 799dc4b83e3f4a14b3ca241d056c527c P 777d8ce7dd4d4bc08ef85971a69083da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:56:24.233150 4979 raft_consensus.cc:2258] T 799dc4b83e3f4a14b3ca241d056c527c P 777d8ce7dd4d4bc08ef85971a69083da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:24.233280 4980 raft_consensus.cc:2258] T 799dc4b83e3f4a14b3ca241d056c527c P 3f4b671c6fa5424cbcd00ffbbe8a392d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:24.233831 4979 ts_tablet_manager.cc:1905] T 799dc4b83e3f4a14b3ca241d056c527c P 777d8ce7dd4d4bc08ef85971a69083da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:24.233942 4980 ts_tablet_manager.cc:1905] T 799dc4b83e3f4a14b3ca241d056c527c P 3f4b671c6fa5424cbcd00ffbbe8a392d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:24.235452 4979 ts_tablet_manager.cc:1918] T 799dc4b83e3f4a14b3ca241d056c527c P 777d8ce7dd4d4bc08ef85971a69083da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:56:24.235507 4980 ts_tablet_manager.cc:1918] T 799dc4b83e3f4a14b3ca241d056c527c P 3f4b671c6fa5424cbcd00ffbbe8a392d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:56:24.235531 4979 log.cc:1196] T 799dc4b83e3f4a14b3ca241d056c527c P 777d8ce7dd4d4bc08ef85971a69083da: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-1/wal/wals/799dc4b83e3f4a14b3ca241d056c527c
I20260619 08:56:24.235566 4980 log.cc:1196] T 799dc4b83e3f4a14b3ca241d056c527c P 3f4b671c6fa5424cbcd00ffbbe8a392d: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859361540516-392-0/minicluster-data/ts-2/wal/wals/799dc4b83e3f4a14b3ca241d056c527c
I20260619 08:56:24.235803 4979 ts_tablet_manager.cc:1939] T 799dc4b83e3f4a14b3ca241d056c527c P 777d8ce7dd4d4bc08ef85971a69083da: Deleting consensus metadata
I20260619 08:56:24.235847 4980 ts_tablet_manager.cc:1939] T 799dc4b83e3f4a14b3ca241d056c527c P 3f4b671c6fa5424cbcd00ffbbe8a392d: Deleting consensus metadata
I20260619 08:56:24.236287 4469 catalog_manager.cc:4841] TS 3f4b671c6fa5424cbcd00ffbbe8a392d (127.0.98.3:42263): tablet 799dc4b83e3f4a14b3ca241d056c527c (table test-table [id=bfea7d86a5ef48f198bfc78f127fc04a]) successfully deleted
I20260619 08:56:24.236411 4471 catalog_manager.cc:4841] TS 777d8ce7dd4d4bc08ef85971a69083da (127.0.98.2:38627): tablet 799dc4b83e3f4a14b3ca241d056c527c (table test-table [id=bfea7d86a5ef48f198bfc78f127fc04a]) successfully deleted
W20260619 08:56:24.237589 4472 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35161 (error 108)
I20260619 08:56:24.238049 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:24.237936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35161 (local address 127.0.0.1:51540)
0619 08:56:24.237986 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:24.238017 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 08:56:24.238137 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 4680
W20260619 08:56:24.238178 4472 catalog_manager.cc:4584] TS 539ed6d90dd74f89981157697fba99cd (127.0.98.1:35161): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 799dc4b83e3f4a14b3ca241d056c527c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35161: connect: Connection refused (error 111)
I20260619 08:56:24.242272 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 4813
I20260619 08:56:24.246088 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 4454
2026-06-19T08:56:24Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1938 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T08:56:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:56:24Z Disabled control of system clock
I20260619 08:56:24.271590 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42297
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 08:56:24.349498 4985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:24.349697 4985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:24.349717 4985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:24.350840 4985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:56:24.350879 4985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:24.350890 4985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:24.350901 4985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:56:24.350914 4985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:56:24.352243 4985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42297
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:24.352465 4985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:24.352730 4985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:24.357161 4993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:24.357259 4990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:24.357906 4991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:24.358050 4985 server_base.cc:1034] running on GCE node
I20260619 08:56:24.358278 4985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:24.358548 4985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:24.359689 4985 hybrid_clock.cc:648] HybridClock initialized: now 1781859384359676 us; error 29 us; skew 500 ppm
I20260619 08:56:24.360893 4985 webserver.cc:460] Webserver started at http://127.0.98.62:33293/ using document root <none> and password file <none>
I20260619 08:56:24.361114 4985 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:24.361164 4985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:24.361300 4985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:24.362325 4985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "bf818973c51f4f63b2693d45d4701057"
format_stamp: "Formatted at 2026-06-19 08:56:24 on dist-test-slave-h4g5"
server_key: "a581cc3467ef3837817e7efb9a4f465b"
server_key_iv: "625347890f89999351daeb2ad954a27e"
server_key_version: "encryptionkey@0"
I20260619 08:56:24.362658 4985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "bf818973c51f4f63b2693d45d4701057"
format_stamp: "Formatted at 2026-06-19 08:56:24 on dist-test-slave-h4g5"
server_key: "a581cc3467ef3837817e7efb9a4f465b"
server_key_iv: "625347890f89999351daeb2ad954a27e"
server_key_version: "encryptionkey@0"
I20260619 08:56:24.363862 4985 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:56:24.364593 4999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:24.364765 4985 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:24.364845 4985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "bf818973c51f4f63b2693d45d4701057"
format_stamp: "Formatted at 2026-06-19 08:56:24 on dist-test-slave-h4g5"
server_key: "a581cc3467ef3837817e7efb9a4f465b"
server_key_iv: "625347890f89999351daeb2ad954a27e"
server_key_version: "encryptionkey@0"
I20260619 08:56:24.364905 4985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:24.368074 4985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:24.368351 4985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:24.368474 4985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:24.371894 4985 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42297
I20260619 08:56:24.371923 5051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42297 every 8 connection(s)
I20260619 08:56:24.372251 4985 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:56:24.372854 5052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:24.375422 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 4985
I20260619 08:56:24.375449 5052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf818973c51f4f63b2693d45d4701057: Bootstrap starting.
I20260619 08:56:24.375506 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 08:56:24.375666 392 external_mini_cluster.cc:1428] Setting key 8fabe61e4dc5121dab5454d1b0656c71
I20260619 08:56:24.376168 5052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf818973c51f4f63b2693d45d4701057: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:24.376474 5052 log.cc:824] T 00000000000000000000000000000000 P bf818973c51f4f63b2693d45d4701057: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:24.377159 5052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf818973c51f4f63b2693d45d4701057: No bootstrap required, opened a new log
I20260619 08:56:24.378038 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:24.376148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38592 (local address 127.0.98.62:42297)
0619 08:56:24.376349 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:24.376359 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:24.376390 (+ 31us) server_negotiation.cc:407] Connection header received
0619 08:56:24.376872 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:24.376881 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:24.376932 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:24.377079 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:24.377624 (+ 545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:24.377627 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:24.377628 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:56:24.377631 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:24.377676 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:56:24.377689 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:24.377777 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:24.377805 (+ 28us) server_negotiation.cc:299] Negotiation successful
0619 08:56:24.377859 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
I20260619 08:56:24.379063 5052 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf818973c51f4f63b2693d45d4701057 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf818973c51f4f63b2693d45d4701057" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42297 } }
I20260619 08:56:24.379186 5052 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf818973c51f4f63b2693d45d4701057 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:24.379207 5052 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf818973c51f4f63b2693d45d4701057 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf818973c51f4f63b2693d45d4701057, State: Initialized, Role: FOLLOWER
I20260619 08:56:24.379314 5052 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf818973c51f4f63b2693d45d4701057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf818973c51f4f63b2693d45d4701057" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42297 } }
I20260619 08:56:24.379388 5052 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf818973c51f4f63b2693d45d4701057 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:56:24.379411 5052 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf818973c51f4f63b2693d45d4701057 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:56:24.379467 5052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf818973c51f4f63b2693d45d4701057 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:24.380290 5052 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf818973c51f4f63b2693d45d4701057 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf818973c51f4f63b2693d45d4701057" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42297 } }
I20260619 08:56:24.380376 5052 leader_election.cc:302] T 00000000000000000000000000000000 P bf818973c51f4f63b2693d45d4701057 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf818973c51f4f63b2693d45d4701057; no voters:
I20260619 08:56:24.380530 5052 leader_election.cc:288] T 00000000000000000000000000000000 P bf818973c51f4f63b2693d45d4701057 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:56:24.380573 5057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf818973c51f4f63b2693d45d4701057 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:24.380733 5057 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf818973c51f4f63b2693d45d4701057 [term 1 LEADER]: Becoming Leader. State: Replica: bf818973c51f4f63b2693d45d4701057, State: Running, Role: LEADER
I20260619 08:56:24.380765 5052 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf818973c51f4f63b2693d45d4701057 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:56:24.380864 5057 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf818973c51f4f63b2693d45d4701057 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf818973c51f4f63b2693d45d4701057" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42297 } }
I20260619 08:56:24.381194 5059 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf818973c51f4f63b2693d45d4701057 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf818973c51f4f63b2693d45d4701057. Latest consensus state: current_term: 1 leader_uuid: "bf818973c51f4f63b2693d45d4701057" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf818973c51f4f63b2693d45d4701057" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42297 } } }
I20260619 08:56:24.381178 5058 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf818973c51f4f63b2693d45d4701057 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf818973c51f4f63b2693d45d4701057" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf818973c51f4f63b2693d45d4701057" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42297 } } }
I20260619 08:56:24.381283 5059 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf818973c51f4f63b2693d45d4701057 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:24.381290 5058 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf818973c51f4f63b2693d45d4701057 [sys.catalog]: This master's current role is: LEADER
W20260619 08:56:24.382385 5073 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bf818973c51f4f63b2693d45d4701057: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 08:56:24.382438 5073 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 08:56:24.382485 5066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:56:24.382616 5066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:56:24.383916 5066 catalog_manager.cc:1347] Generated new cluster ID: d55e97f89cad4379943dec8adcbb6a5a
I20260619 08:56:24.383967 5066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:56:24.400058 5066 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:56:24.400262 5066 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:56:24.407317 5066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf818973c51f4f63b2693d45d4701057: Generated new TSK 0
I20260619 08:56:24.407497 5066 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:56:24.410302 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42297
--builtin_ntp_servers=127.0.98.20:35057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 08:56:24.487764 5076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:24.487959 5076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:24.487979 5076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:24.489127 5076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:24.489166 5076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:24.489190 5076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 08:56:24.490485 5076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:24.490697 5076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:24.490963 5076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:24.495586 5082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:24.495591 5081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:24.495611 5084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:24.495780 5076 server_base.cc:1034] running on GCE node
I20260619 08:56:24.496016 5076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:24.496243 5076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:24.497392 5076 hybrid_clock.cc:648] HybridClock initialized: now 1781859384497375 us; error 29 us; skew 500 ppm
I20260619 08:56:24.498579 5076 webserver.cc:460] Webserver started at http://127.0.98.1:43033/ using document root <none> and password file <none>
I20260619 08:56:24.498791 5076 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:24.498838 5076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:24.498976 5076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:24.500003 5076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-0/data/instance:
uuid: "eb9635c71d4d41b28881a803fc9ad0f3"
format_stamp: "Formatted at 2026-06-19 08:56:24 on dist-test-slave-h4g5"
server_key: "67c25f6d80723ae541d6724234dbeafb"
server_key_iv: "983bcd96a0fc0bd4c1b051cbd5122242"
server_key_version: "encryptionkey@0"
I20260619 08:56:24.500380 5076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-0/wal/instance:
uuid: "eb9635c71d4d41b28881a803fc9ad0f3"
format_stamp: "Formatted at 2026-06-19 08:56:24 on dist-test-slave-h4g5"
server_key: "67c25f6d80723ae541d6724234dbeafb"
server_key_iv: "983bcd96a0fc0bd4c1b051cbd5122242"
server_key_version: "encryptionkey@0"
I20260619 08:56:24.501778 5076 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:24.502451 5090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:24.502621 5076 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:24.502700 5076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-0/wal
uuid: "eb9635c71d4d41b28881a803fc9ad0f3"
format_stamp: "Formatted at 2026-06-19 08:56:24 on dist-test-slave-h4g5"
server_key: "67c25f6d80723ae541d6724234dbeafb"
server_key_iv: "983bcd96a0fc0bd4c1b051cbd5122242"
server_key_version: "encryptionkey@0"
I20260619 08:56:24.502768 5076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:24.506150 5076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:24.506394 5076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:24.506510 5076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:24.506699 5076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:24.506981 5076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:24.507014 5076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:24.507038 5076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:24.507062 5076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:24.512428 5076 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42463
I20260619 08:56:24.512465 5203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42463 every 8 connection(s)
I20260619 08:56:24.512761 5076 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
I20260619 08:56:24.514377 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 5076
I20260619 08:56:24.514460 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-0/wal/instance
I20260619 08:56:24.514643 392 external_mini_cluster.cc:1428] Setting key 4de87547aa5810cf6bfc58681ef1c0d1
I20260619 08:56:24.515640 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42297
--builtin_ntp_servers=127.0.98.20:35057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 08:56:24.515883 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:24.513419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57845 (local address 127.0.98.62:42297)
0619 08:56:24.513460 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:24.513462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:24.513698 (+ 236us) server_negotiation.cc:407] Connection header received
0619 08:56:24.514182 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:24.514183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:24.514190 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:24.514273 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:24.515559 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:24.515560 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:24.515561 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:56:24.515565 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:24.515582 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:56:24.515586 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:24.515644 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:24.515808 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 08:56:24.515823 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 08:56:24.515951 5206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:24.513403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42297 (local address 127.0.98.1:57845)
0619 08:56:24.513583 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:24.513616 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:24.514085 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:24.514347 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:24.514350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:24.514358 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:24.515404 (+ 1046us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:56:24.515405 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:24.515442 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:56:24.515474 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:24.515643 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:24.515645 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:24.515647 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:56:24.515702 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 08:56:24.515764 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
I20260619 08:56:24.516372 5204 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42297
I20260619 08:56:24.516463 5204 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:24.516598 5204 heartbeater.cc:511] Master 127.0.98.62:42297 requested a full tablet report, sending...
I20260619 08:56:24.517071 5016 ts_manager.cc:194] Registered new tserver with Master: eb9635c71d4d41b28881a803fc9ad0f3 (127.0.98.1:42463)
W20260619 08:56:24.596763 5207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:24.596954 5207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:24.596974 5207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:24.598043 5207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:24.598075 5207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:24.598099 5207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 08:56:24.599413 5207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:24.599599 5207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:24.599838 5207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:24.604493 5212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:24.604486 5213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:24.604486 5215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:24.604514 5207 server_base.cc:1034] running on GCE node
I20260619 08:56:24.604988 5207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:24.605239 5207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:24.606382 5207 hybrid_clock.cc:648] HybridClock initialized: now 1781859384606367 us; error 27 us; skew 500 ppm
I20260619 08:56:24.607645 5207 webserver.cc:460] Webserver started at http://127.0.98.2:37751/ using document root <none> and password file <none>
I20260619 08:56:24.607867 5207 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:24.607916 5207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:24.608037 5207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:24.609092 5207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-1/data/instance:
uuid: "0bcbfe6c8cf5486dabdbe0b92c6c1264"
format_stamp: "Formatted at 2026-06-19 08:56:24 on dist-test-slave-h4g5"
server_key: "2c8efd6a79aaaf85aeb838e000c5fe55"
server_key_iv: "a847cf110cbffc60108fc5f4c3c6ff76"
server_key_version: "encryptionkey@0"
I20260619 08:56:24.609408 5207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0bcbfe6c8cf5486dabdbe0b92c6c1264"
format_stamp: "Formatted at 2026-06-19 08:56:24 on dist-test-slave-h4g5"
server_key: "2c8efd6a79aaaf85aeb838e000c5fe55"
server_key_iv: "a847cf110cbffc60108fc5f4c3c6ff76"
server_key_version: "encryptionkey@0"
I20260619 08:56:24.610656 5207 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:56:24.611256 5221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:24.611406 5207 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:24.611483 5207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-1/wal
uuid: "0bcbfe6c8cf5486dabdbe0b92c6c1264"
format_stamp: "Formatted at 2026-06-19 08:56:24 on dist-test-slave-h4g5"
server_key: "2c8efd6a79aaaf85aeb838e000c5fe55"
server_key_iv: "a847cf110cbffc60108fc5f4c3c6ff76"
server_key_version: "encryptionkey@0"
I20260619 08:56:24.611541 5207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:24.614799 5207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:24.615032 5207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:24.615152 5207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:24.615322 5207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:24.615599 5207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:24.615633 5207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:24.615657 5207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:24.615682 5207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:24.620936 5207 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33929
I20260619 08:56:24.620988 5334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33929 every 8 connection(s)
I20260619 08:56:24.621263 5207 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
I20260619 08:56:24.623878 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:24.621895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33711 (local address 127.0.98.62:42297)
0619 08:56:24.621936 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:24.621938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:24.622152 (+ 214us) server_negotiation.cc:407] Connection header received
0619 08:56:24.622592 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:24.622592 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:24.622598 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:24.622647 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:24.623715 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:24.623716 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:24.623716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:56:24.623718 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:24.623728 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:56:24.623731 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:24.623765 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:24.623814 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 08:56:24.623825 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 08:56:24.624063 5337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:24.621884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42297 (local address 127.0.98.2:33711)
0619 08:56:24.622031 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:24.622065 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:24.622501 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:24.622701 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:24.622704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:24.622713 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:24.623591 (+ 878us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:56:24.623591 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:24.623621 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:56:24.623649 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:24.623770 (+ 121us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:24.623773 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:24.623775 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:56:24.623822 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 08:56:24.623867 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":65,"threads_started":1}
I20260619 08:56:24.624480 5335 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42297
I20260619 08:56:24.624565 5335 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:24.624706 5335 heartbeater.cc:511] Master 127.0.98.62:42297 requested a full tablet report, sending...
I20260619 08:56:24.625078 5016 ts_manager.cc:194] Registered new tserver with Master: 0bcbfe6c8cf5486dabdbe0b92c6c1264 (127.0.98.2:33929)
I20260619 08:56:24.630398 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 5207
I20260619 08:56:24.630498 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-1/wal/instance
I20260619 08:56:24.630656 392 external_mini_cluster.cc:1428] Setting key 06a4d740538085af849212ca2aefd47f
I20260619 08:56:24.631749 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42297
--builtin_ntp_servers=127.0.98.20:35057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 08:56:24.715063 5338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:24.715277 5338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:24.715305 5338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:24.716478 5338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:24.716517 5338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:24.716552 5338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 08:56:24.717908 5338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:24.718153 5338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:24.718427 5338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:24.723484 5343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:24.723454 5344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:24.723477 5338 server_base.cc:1034] running on GCE node
W20260619 08:56:24.723461 5346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:24.723928 5338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:24.724171 5338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:24.725322 5338 hybrid_clock.cc:648] HybridClock initialized: now 1781859384725304 us; error 34 us; skew 500 ppm
I20260619 08:56:24.726574 5338 webserver.cc:460] Webserver started at http://127.0.98.3:41563/ using document root <none> and password file <none>
I20260619 08:56:24.726820 5338 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:24.726867 5338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:24.726980 5338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:24.728030 5338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-2/data/instance:
uuid: "2a41c90fd655498f939d826eacb0be02"
format_stamp: "Formatted at 2026-06-19 08:56:24 on dist-test-slave-h4g5"
server_key: "cf4110dc2583b07ee04e08a43559db6c"
server_key_iv: "12bc8b81eb1631a6c61fb83f544b3004"
server_key_version: "encryptionkey@0"
I20260619 08:56:24.728415 5338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2a41c90fd655498f939d826eacb0be02"
format_stamp: "Formatted at 2026-06-19 08:56:24 on dist-test-slave-h4g5"
server_key: "cf4110dc2583b07ee04e08a43559db6c"
server_key_iv: "12bc8b81eb1631a6c61fb83f544b3004"
server_key_version: "encryptionkey@0"
I20260619 08:56:24.729738 5338 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260619 08:56:24.730464 5352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:24.730647 5338 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:24.730752 5338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-2/wal
uuid: "2a41c90fd655498f939d826eacb0be02"
format_stamp: "Formatted at 2026-06-19 08:56:24 on dist-test-slave-h4g5"
server_key: "cf4110dc2583b07ee04e08a43559db6c"
server_key_iv: "12bc8b81eb1631a6c61fb83f544b3004"
server_key_version: "encryptionkey@0"
I20260619 08:56:24.730810 5338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:24.734071 5338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:24.734287 5338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:24.734376 5338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:24.734526 5338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:24.734784 5338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:24.734813 5338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:24.734831 5338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:24.734848 5338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:24.740003 5338 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38587
I20260619 08:56:24.740062 5465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38587 every 8 connection(s)
I20260619 08:56:24.740355 5338 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
I20260619 08:56:24.743019 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:24.740994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51377 (local address 127.0.98.62:42297)
0619 08:56:24.741050 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:24.741052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:24.741276 (+ 224us) server_negotiation.cc:407] Connection header received
0619 08:56:24.741730 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:24.741731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:24.741736 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:24.741779 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:24.742798 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:24.742799 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:24.742800 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:56:24.742803 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:24.742817 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:56:24.742821 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:24.742864 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:24.742948 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 08:56:24.742962 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 08:56:24.743134 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:24.740952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42297 (local address 127.0.98.3:51377)
0619 08:56:24.741135 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:24.741173 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:24.741633 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:24.741838 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:24.741841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:24.741849 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:24.742661 (+ 812us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:56:24.742661 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:24.742689 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:56:24.742717 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:24.742864 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:24.742866 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:24.742869 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:56:24.742919 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 08:56:24.742965 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260619 08:56:24.743556 5466 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42297
I20260619 08:56:24.743631 5466 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:24.743764 5466 heartbeater.cc:511] Master 127.0.98.62:42297 requested a full tablet report, sending...
I20260619 08:56:24.744064 5016 ts_manager.cc:194] Registered new tserver with Master: 2a41c90fd655498f939d826eacb0be02 (127.0.98.3:38587)
I20260619 08:56:24.746358 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 5338
I20260619 08:56:24.746452 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-2/wal/instance
I20260619 08:56:24.746632 392 external_mini_cluster.cc:1428] Setting key e56b3af60fa99a54ca64228e1f73f146
I20260619 08:56:24.747445 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:56:24.749531 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:24.748667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38596 (local address 127.0.98.62:42297)
0619 08:56:24.748717 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:24.748718 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:24.748804 (+ 86us) server_negotiation.cc:407] Connection header received
0619 08:56:24.748854 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:24.748854 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:24.748858 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:24.748900 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:24.749383 (+ 483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:24.749384 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:24.749384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:56:24.749387 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:24.749400 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:56:24.749403 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:24.749444 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:24.749475 (+ 31us) server_negotiation.cc:299] Negotiation successful
0619 08:56:24.749485 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 08:56:24.750948 5016 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:56:24.752621 5016 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:56:24.756255 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:24.754528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42463 (local address 127.0.0.1:34496)
0619 08:56:24.754796 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:24.754835 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:24.754888 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:24.755376 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:24.755378 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:24.755381 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:24.755831 (+ 450us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:56:24.755832 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:24.755847 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:56:24.755870 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:24.756069 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:24.756074 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:24.756104 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 08:56:24.756174 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 08:56:24.756187 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":63,"threads_started":1}
I20260619 08:56:24.756261 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:24.754617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34496 (local address 127.0.98.1:42463)
0619 08:56:24.754944 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:24.754948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:24.754958 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:56:24.754981 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:24.754983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:24.754990 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:24.755053 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:24.755938 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:24.755947 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:24.755948 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:56:24.755952 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:24.755986 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:56:24.756016 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:24.756076 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:24.756188 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 08:56:24.756202 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":89,"threads_started":1}
I20260619 08:56:24.756361 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:24.754537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33929 (local address 127.0.0.1:38574)
0619 08:56:24.754832 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:24.754840 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:24.754889 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:24.755362 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:24.755368 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:24.755374 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:24.756002 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:56:24.756003 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:24.756008 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:56:24.756017 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:24.756260 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:24.756261 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:24.756261 (+ 0us) client_negotiation.cc:769] Sending connection context
0619 08:56:24.756301 (+ 40us) client_negotiation.cc:240] Negotiation successful
0619 08:56:24.756311 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":95,"threads_started":1}
I20260619 08:56:24.756580 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:24.754614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38574 (local address 127.0.98.2:33929)
0619 08:56:24.754744 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:24.754748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:24.754948 (+ 200us) server_negotiation.cc:407] Connection header received
0619 08:56:24.754977 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:24.754980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:24.754990 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:24.755066 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:24.756120 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:24.756128 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:24.756128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:56:24.756132 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:24.756166 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:56:24.756189 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:24.756238 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:24.756488 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 08:56:24.756505 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":70,"threads_started":1}
I20260619 08:56:24.756809 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:24.754725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38587 (local address 127.0.0.1:38840)
0619 08:56:24.755566 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:24.755574 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:24.755617 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:24.755763 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:24.755764 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:24.755766 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:24.756442 (+ 676us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:56:24.756443 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:24.756448 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:56:24.756459 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:24.756693 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:24.756693 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:24.756694 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 08:56:24.756744 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 08:56:24.756756 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"thread_start_us":69,"threads_started":1}
I20260619 08:56:24.756928 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:24.754917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38840 (local address 127.0.98.3:38587)
0619 08:56:24.755074 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:24.755078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:24.755621 (+ 543us) server_negotiation.cc:407] Connection header received
0619 08:56:24.755668 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:24.755670 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:24.755678 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:24.755748 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:24.756539 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:24.756545 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:24.756545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:56:24.756549 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:24.756581 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:56:24.756615 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:24.756669 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:24.756772 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 08:56:24.756788 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
I20260619 08:56:24.756923 5138 tablet_service.cc:1467] Processing CreateTablet for tablet 686ddb12789341e4b93aa62ef5d35a9d (DEFAULT_TABLE table=test-table [id=f7dceb86d5dd4398b55b51264841ed12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:24.757313 5138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 686ddb12789341e4b93aa62ef5d35a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:24.759691 5480 tablet_bootstrap.cc:492] T 686ddb12789341e4b93aa62ef5d35a9d P eb9635c71d4d41b28881a803fc9ad0f3: Bootstrap starting.
I20260619 08:56:24.759985 5269 tablet_service.cc:1467] Processing CreateTablet for tablet 686ddb12789341e4b93aa62ef5d35a9d (DEFAULT_TABLE table=test-table [id=f7dceb86d5dd4398b55b51264841ed12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:24.760293 5269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 686ddb12789341e4b93aa62ef5d35a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:24.760404 5480 tablet_bootstrap.cc:654] T 686ddb12789341e4b93aa62ef5d35a9d P eb9635c71d4d41b28881a803fc9ad0f3: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:24.760479 5400 tablet_service.cc:1467] Processing CreateTablet for tablet 686ddb12789341e4b93aa62ef5d35a9d (DEFAULT_TABLE table=test-table [id=f7dceb86d5dd4398b55b51264841ed12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:24.760767 5480 log.cc:824] T 686ddb12789341e4b93aa62ef5d35a9d P eb9635c71d4d41b28881a803fc9ad0f3: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:24.760784 5400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 686ddb12789341e4b93aa62ef5d35a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:24.761510 5480 tablet_bootstrap.cc:492] T 686ddb12789341e4b93aa62ef5d35a9d P eb9635c71d4d41b28881a803fc9ad0f3: No bootstrap required, opened a new log
I20260619 08:56:24.761590 5480 ts_tablet_manager.cc:1397] T 686ddb12789341e4b93aa62ef5d35a9d P eb9635c71d4d41b28881a803fc9ad0f3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:56:24.762828 5482 tablet_bootstrap.cc:492] T 686ddb12789341e4b93aa62ef5d35a9d P 0bcbfe6c8cf5486dabdbe0b92c6c1264: Bootstrap starting.
I20260619 08:56:24.763423 5482 tablet_bootstrap.cc:654] T 686ddb12789341e4b93aa62ef5d35a9d P 0bcbfe6c8cf5486dabdbe0b92c6c1264: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:24.763703 5482 log.cc:824] T 686ddb12789341e4b93aa62ef5d35a9d P 0bcbfe6c8cf5486dabdbe0b92c6c1264: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:24.763932 5483 tablet_bootstrap.cc:492] T 686ddb12789341e4b93aa62ef5d35a9d P 2a41c90fd655498f939d826eacb0be02: Bootstrap starting.
I20260619 08:56:24.764035 5480 raft_consensus.cc:348] T 686ddb12789341e4b93aa62ef5d35a9d P eb9635c71d4d41b28881a803fc9ad0f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bcbfe6c8cf5486dabdbe0b92c6c1264" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33929 } } peers { permanent_uuid: "eb9635c71d4d41b28881a803fc9ad0f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42463 } } peers { permanent_uuid: "2a41c90fd655498f939d826eacb0be02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38587 } }
I20260619 08:56:24.764226 5480 raft_consensus.cc:374] T 686ddb12789341e4b93aa62ef5d35a9d P eb9635c71d4d41b28881a803fc9ad0f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:24.764292 5480 raft_consensus.cc:729] T 686ddb12789341e4b93aa62ef5d35a9d P eb9635c71d4d41b28881a803fc9ad0f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb9635c71d4d41b28881a803fc9ad0f3, State: Initialized, Role: FOLLOWER
I20260619 08:56:24.764418 5480 consensus_queue.cc:260] T 686ddb12789341e4b93aa62ef5d35a9d P eb9635c71d4d41b28881a803fc9ad0f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bcbfe6c8cf5486dabdbe0b92c6c1264" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33929 } } peers { permanent_uuid: "eb9635c71d4d41b28881a803fc9ad0f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42463 } } peers { permanent_uuid: "2a41c90fd655498f939d826eacb0be02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38587 } }
I20260619 08:56:24.764590 5483 tablet_bootstrap.cc:654] T 686ddb12789341e4b93aa62ef5d35a9d P 2a41c90fd655498f939d826eacb0be02: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:24.764672 5480 ts_tablet_manager.cc:1428] T 686ddb12789341e4b93aa62ef5d35a9d P eb9635c71d4d41b28881a803fc9ad0f3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:56:24.764699 5482 tablet_bootstrap.cc:492] T 686ddb12789341e4b93aa62ef5d35a9d P 0bcbfe6c8cf5486dabdbe0b92c6c1264: No bootstrap required, opened a new log
I20260619 08:56:24.764784 5204 heartbeater.cc:503] Master 127.0.98.62:42297 was elected leader, sending a full tablet report...
I20260619 08:56:24.764819 5482 ts_tablet_manager.cc:1397] T 686ddb12789341e4b93aa62ef5d35a9d P 0bcbfe6c8cf5486dabdbe0b92c6c1264: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:56:24.764881 5483 log.cc:824] T 686ddb12789341e4b93aa62ef5d35a9d P 2a41c90fd655498f939d826eacb0be02: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:24.765611 5483 tablet_bootstrap.cc:492] T 686ddb12789341e4b93aa62ef5d35a9d P 2a41c90fd655498f939d826eacb0be02: No bootstrap required, opened a new log
I20260619 08:56:24.765676 5483 ts_tablet_manager.cc:1397] T 686ddb12789341e4b93aa62ef5d35a9d P 2a41c90fd655498f939d826eacb0be02: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:56:24.767248 5482 raft_consensus.cc:348] T 686ddb12789341e4b93aa62ef5d35a9d P 0bcbfe6c8cf5486dabdbe0b92c6c1264 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bcbfe6c8cf5486dabdbe0b92c6c1264" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33929 } } peers { permanent_uuid: "eb9635c71d4d41b28881a803fc9ad0f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42463 } } peers { permanent_uuid: "2a41c90fd655498f939d826eacb0be02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38587 } }
I20260619 08:56:24.767486 5482 raft_consensus.cc:374] T 686ddb12789341e4b93aa62ef5d35a9d P 0bcbfe6c8cf5486dabdbe0b92c6c1264 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:24.767556 5482 raft_consensus.cc:729] T 686ddb12789341e4b93aa62ef5d35a9d P 0bcbfe6c8cf5486dabdbe0b92c6c1264 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bcbfe6c8cf5486dabdbe0b92c6c1264, State: Initialized, Role: FOLLOWER
I20260619 08:56:24.767745 5482 consensus_queue.cc:260] T 686ddb12789341e4b93aa62ef5d35a9d P 0bcbfe6c8cf5486dabdbe0b92c6c1264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bcbfe6c8cf5486dabdbe0b92c6c1264" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33929 } } peers { permanent_uuid: "eb9635c71d4d41b28881a803fc9ad0f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42463 } } peers { permanent_uuid: "2a41c90fd655498f939d826eacb0be02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38587 } }
I20260619 08:56:24.767993 5482 ts_tablet_manager.cc:1428] T 686ddb12789341e4b93aa62ef5d35a9d P 0bcbfe6c8cf5486dabdbe0b92c6c1264: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:56:24.768030 5335 heartbeater.cc:503] Master 127.0.98.62:42297 was elected leader, sending a full tablet report...
I20260619 08:56:24.768021 5483 raft_consensus.cc:348] T 686ddb12789341e4b93aa62ef5d35a9d P 2a41c90fd655498f939d826eacb0be02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bcbfe6c8cf5486dabdbe0b92c6c1264" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33929 } } peers { permanent_uuid: "eb9635c71d4d41b28881a803fc9ad0f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42463 } } peers { permanent_uuid: "2a41c90fd655498f939d826eacb0be02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38587 } }
I20260619 08:56:24.768199 5483 raft_consensus.cc:374] T 686ddb12789341e4b93aa62ef5d35a9d P 2a41c90fd655498f939d826eacb0be02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:24.768270 5483 raft_consensus.cc:729] T 686ddb12789341e4b93aa62ef5d35a9d P 2a41c90fd655498f939d826eacb0be02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a41c90fd655498f939d826eacb0be02, State: Initialized, Role: FOLLOWER
I20260619 08:56:24.768416 5483 consensus_queue.cc:260] T 686ddb12789341e4b93aa62ef5d35a9d P 2a41c90fd655498f939d826eacb0be02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bcbfe6c8cf5486dabdbe0b92c6c1264" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33929 } } peers { permanent_uuid: "eb9635c71d4d41b28881a803fc9ad0f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42463 } } peers { permanent_uuid: "2a41c90fd655498f939d826eacb0be02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38587 } }
I20260619 08:56:24.768653 5483 ts_tablet_manager.cc:1428] T 686ddb12789341e4b93aa62ef5d35a9d P 2a41c90fd655498f939d826eacb0be02: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:56:24.768772 5466 heartbeater.cc:503] Master 127.0.98.62:42297 was elected leader, sending a full tablet report...
W20260619 08:56:24.871775 5336 tablet.cc:2369] T 686ddb12789341e4b93aa62ef5d35a9d P 0bcbfe6c8cf5486dabdbe0b92c6c1264: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 08:56:24.990893 5467 tablet.cc:2369] T 686ddb12789341e4b93aa62ef5d35a9d P 2a41c90fd655498f939d826eacb0be02: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 08:56:25.013451 5205 tablet.cc:2369] T 686ddb12789341e4b93aa62ef5d35a9d P eb9635c71d4d41b28881a803fc9ad0f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:56:25.053586 5488 raft_consensus.cc:482] T 686ddb12789341e4b93aa62ef5d35a9d P 2a41c90fd655498f939d826eacb0be02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:56:25.053722 5488 raft_consensus.cc:504] T 686ddb12789341e4b93aa62ef5d35a9d P 2a41c90fd655498f939d826eacb0be02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bcbfe6c8cf5486dabdbe0b92c6c1264" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33929 } } peers { permanent_uuid: "eb9635c71d4d41b28881a803fc9ad0f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42463 } } peers { permanent_uuid: "2a41c90fd655498f939d826eacb0be02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38587 } }
I20260619 08:56:25.054028 5488 leader_election.cc:288] T 686ddb12789341e4b93aa62ef5d35a9d P 2a41c90fd655498f939d826eacb0be02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bcbfe6c8cf5486dabdbe0b92c6c1264 (127.0.98.2:33929), eb9635c71d4d41b28881a803fc9ad0f3 (127.0.98.1:42463)
I20260619 08:56:25.055371 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:25.054296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52251 (local address 127.0.98.2:33929)
0619 08:56:25.054344 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:25.054346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:25.054398 (+ 52us) server_negotiation.cc:407] Connection header received
0619 08:56:25.054424 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:25.054424 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:25.054429 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:25.054482 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:25.055093 (+ 611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:25.055094 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:25.055095 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:56:25.055100 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:25.055117 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:56:25.055121 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:25.055179 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:25.055289 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 08:56:25.055302 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 08:56:25.055371 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:25.054338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47159 (local address 127.0.98.1:42463)
0619 08:56:25.054377 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:25.054378 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:25.054459 (+ 81us) server_negotiation.cc:407] Connection header received
0619 08:56:25.054503 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:25.054504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:25.054509 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:25.054552 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:25.055161 (+ 609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:25.055162 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:25.055163 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:56:25.055166 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:25.055181 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:56:25.055185 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:25.055237 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:25.055285 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 08:56:25.055298 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 08:56:25.055372 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:25.054230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33929 (local address 127.0.98.3:52251)
0619 08:56:25.054284 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:25.054296 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:25.054337 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:25.054499 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:25.054500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:25.054502 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:25.054978 (+ 476us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:56:25.054978 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:25.054985 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:56:25.054999 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:25.055188 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:25.055188 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:25.055189 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 08:56:25.055273 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 08:56:25.055287 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 08:56:25.055382 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:25.054230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42463 (local address 127.0.98.3:47159)
0619 08:56:25.054399 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:25.054406 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:25.054454 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:25.054568 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:25.054569 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:25.054571 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:25.055068 (+ 497us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:56:25.055068 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:25.055072 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:56:25.055081 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:25.055239 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:25.055240 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:25.055240 (+ 0us) client_negotiation.cc:769] Sending connection context
0619 08:56:25.055285 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 08:56:25.055293 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":15,"thread_start_us":95,"threads_started":1}
I20260619 08:56:25.055728 5289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "686ddb12789341e4b93aa62ef5d35a9d" candidate_uuid: "2a41c90fd655498f939d826eacb0be02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bcbfe6c8cf5486dabdbe0b92c6c1264" is_pre_election: true
I20260619 08:56:25.055704 5158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "686ddb12789341e4b93aa62ef5d35a9d" candidate_uuid: "2a41c90fd655498f939d826eacb0be02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb9635c71d4d41b28881a803fc9ad0f3" is_pre_election: true
I20260619 08:56:25.055881 5289 raft_consensus.cc:2454] T 686ddb12789341e4b93aa62ef5d35a9d P 0bcbfe6c8cf5486dabdbe0b92c6c1264 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a41c90fd655498f939d826eacb0be02 in term 0.
I20260619 08:56:25.055881 5158 raft_consensus.cc:2454] T 686ddb12789341e4b93aa62ef5d35a9d P eb9635c71d4d41b28881a803fc9ad0f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a41c90fd655498f939d826eacb0be02 in term 0.
I20260619 08:56:25.056139 5355 leader_election.cc:302] T 686ddb12789341e4b93aa62ef5d35a9d P 2a41c90fd655498f939d826eacb0be02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a41c90fd655498f939d826eacb0be02, eb9635c71d4d41b28881a803fc9ad0f3; no voters:
I20260619 08:56:25.056319 5488 raft_consensus.cc:2781] T 686ddb12789341e4b93aa62ef5d35a9d P 2a41c90fd655498f939d826eacb0be02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:56:25.056377 5488 raft_consensus.cc:482] T 686ddb12789341e4b93aa62ef5d35a9d P 2a41c90fd655498f939d826eacb0be02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:56:25.056392 5488 raft_consensus.cc:3037] T 686ddb12789341e4b93aa62ef5d35a9d P 2a41c90fd655498f939d826eacb0be02 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:25.057031 5488 raft_consensus.cc:504] T 686ddb12789341e4b93aa62ef5d35a9d P 2a41c90fd655498f939d826eacb0be02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bcbfe6c8cf5486dabdbe0b92c6c1264" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33929 } } peers { permanent_uuid: "eb9635c71d4d41b28881a803fc9ad0f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42463 } } peers { permanent_uuid: "2a41c90fd655498f939d826eacb0be02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38587 } }
I20260619 08:56:25.057142 5488 leader_election.cc:288] T 686ddb12789341e4b93aa62ef5d35a9d P 2a41c90fd655498f939d826eacb0be02 [CANDIDATE]: Term 1 election: Requested vote from peers 0bcbfe6c8cf5486dabdbe0b92c6c1264 (127.0.98.2:33929), eb9635c71d4d41b28881a803fc9ad0f3 (127.0.98.1:42463)
I20260619 08:56:25.057318 5158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "686ddb12789341e4b93aa62ef5d35a9d" candidate_uuid: "2a41c90fd655498f939d826eacb0be02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb9635c71d4d41b28881a803fc9ad0f3"
I20260619 08:56:25.057337 5289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "686ddb12789341e4b93aa62ef5d35a9d" candidate_uuid: "2a41c90fd655498f939d826eacb0be02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bcbfe6c8cf5486dabdbe0b92c6c1264"
I20260619 08:56:25.057399 5158 raft_consensus.cc:3037] T 686ddb12789341e4b93aa62ef5d35a9d P eb9635c71d4d41b28881a803fc9ad0f3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:25.057408 5289 raft_consensus.cc:3037] T 686ddb12789341e4b93aa62ef5d35a9d P 0bcbfe6c8cf5486dabdbe0b92c6c1264 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:25.058070 5158 raft_consensus.cc:2454] T 686ddb12789341e4b93aa62ef5d35a9d P eb9635c71d4d41b28881a803fc9ad0f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a41c90fd655498f939d826eacb0be02 in term 1.
I20260619 08:56:25.058121 5289 raft_consensus.cc:2454] T 686ddb12789341e4b93aa62ef5d35a9d P 0bcbfe6c8cf5486dabdbe0b92c6c1264 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a41c90fd655498f939d826eacb0be02 in term 1.
I20260619 08:56:25.058233 5355 leader_election.cc:302] T 686ddb12789341e4b93aa62ef5d35a9d P 2a41c90fd655498f939d826eacb0be02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a41c90fd655498f939d826eacb0be02, eb9635c71d4d41b28881a803fc9ad0f3; no voters:
I20260619 08:56:25.058313 5488 raft_consensus.cc:2781] T 686ddb12789341e4b93aa62ef5d35a9d P 2a41c90fd655498f939d826eacb0be02 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:25.058439 5488 raft_consensus.cc:686] T 686ddb12789341e4b93aa62ef5d35a9d P 2a41c90fd655498f939d826eacb0be02 [term 1 LEADER]: Becoming Leader. State: Replica: 2a41c90fd655498f939d826eacb0be02, State: Running, Role: LEADER
I20260619 08:56:25.058516 5488 consensus_queue.cc:237] T 686ddb12789341e4b93aa62ef5d35a9d P 2a41c90fd655498f939d826eacb0be02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bcbfe6c8cf5486dabdbe0b92c6c1264" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33929 } } peers { permanent_uuid: "eb9635c71d4d41b28881a803fc9ad0f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42463 } } peers { permanent_uuid: "2a41c90fd655498f939d826eacb0be02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38587 } }
I20260619 08:56:25.059276 5015 catalog_manager.cc:5495] T 686ddb12789341e4b93aa62ef5d35a9d P 2a41c90fd655498f939d826eacb0be02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a41c90fd655498f939d826eacb0be02 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2a41c90fd655498f939d826eacb0be02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bcbfe6c8cf5486dabdbe0b92c6c1264" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb9635c71d4d41b28881a803fc9ad0f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a41c90fd655498f939d826eacb0be02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38587 } health_report { overall_health: HEALTHY } } }
I20260619 08:56:25.110837 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:25.110006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38852 (local address 127.0.98.3:38587)
0619 08:56:25.110090 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:25.110092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:25.110099 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:56:25.110133 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:25.110133 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:25.110146 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:25.110201 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:25.110651 (+ 450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:25.110652 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:25.110653 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:56:25.110656 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:25.110669 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:56:25.110673 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:25.110717 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:25.110757 (+ 40us) server_negotiation.cc:299] Negotiation successful
0619 08:56:25.110768 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 08:56:25.112608 5289 raft_consensus.cc:1261] T 686ddb12789341e4b93aa62ef5d35a9d P 0bcbfe6c8cf5486dabdbe0b92c6c1264 [term 1 FOLLOWER]: Refusing update from remote peer 2a41c90fd655498f939d826eacb0be02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:56:25.112928 5158 raft_consensus.cc:1261] T 686ddb12789341e4b93aa62ef5d35a9d P eb9635c71d4d41b28881a803fc9ad0f3 [term 1 FOLLOWER]: Refusing update from remote peer 2a41c90fd655498f939d826eacb0be02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:56:25.112947 5490 consensus_queue.cc:1035] T 686ddb12789341e4b93aa62ef5d35a9d P 2a41c90fd655498f939d826eacb0be02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bcbfe6c8cf5486dabdbe0b92c6c1264" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:56:25.113196 5490 consensus_queue.cc:1035] T 686ddb12789341e4b93aa62ef5d35a9d P 2a41c90fd655498f939d826eacb0be02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb9635c71d4d41b28881a803fc9ad0f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:56:25.115248 5493 mvcc.cc:205] Tried to move back new op lower bound from 7298496041418330112 to 7298496041200099328. Current Snapshot: MvccSnapshot[applied={T|T < 7298496041418330112}]
I20260619 08:56:25.116351 5496 mvcc.cc:205] Tried to move back new op lower bound from 7298496041418330112 to 7298496041200099328. Current Snapshot: MvccSnapshot[applied={T|T < 7298496041418330112 or (T in {7298496041418330112})}]
I20260619 08:56:25.117722 5015 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:56:25.117810 5015 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:56:25.118606 5015 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf818973c51f4f63b2693d45d4701057: Sending DeleteTablet for 3 replicas of tablet 686ddb12789341e4b93aa62ef5d35a9d
I20260619 08:56:25.118932 5138 tablet_service.cc:1514] Processing DeleteTablet for tablet 686ddb12789341e4b93aa62ef5d35a9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:25 UTC) from {username='slave'} at 127.0.0.1:34496
I20260619 08:56:25.118944 5400 tablet_service.cc:1514] Processing DeleteTablet for tablet 686ddb12789341e4b93aa62ef5d35a9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:25 UTC) from {username='slave'} at 127.0.0.1:38840
I20260619 08:56:25.119168 5503 tablet_replica.cc:330] T 686ddb12789341e4b93aa62ef5d35a9d P eb9635c71d4d41b28881a803fc9ad0f3: stopping tablet replica
I20260619 08:56:25.119172 5269 tablet_service.cc:1514] Processing DeleteTablet for tablet 686ddb12789341e4b93aa62ef5d35a9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:25 UTC) from {username='slave'} at 127.0.0.1:38574
I20260619 08:56:25.119274 5503 raft_consensus.cc:2229] T 686ddb12789341e4b93aa62ef5d35a9d P eb9635c71d4d41b28881a803fc9ad0f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:56:25.119338 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 5076
I20260619 08:56:25.119395 5503 raft_consensus.cc:2258] T 686ddb12789341e4b93aa62ef5d35a9d P eb9635c71d4d41b28881a803fc9ad0f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:25.119859 5504 tablet_replica.cc:330] T 686ddb12789341e4b93aa62ef5d35a9d P 2a41c90fd655498f939d826eacb0be02: stopping tablet replica
I20260619 08:56:25.119948 5504 raft_consensus.cc:2229] T 686ddb12789341e4b93aa62ef5d35a9d P 2a41c90fd655498f939d826eacb0be02 [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:56:25.120024 5504 raft_consensus.cc:2258] T 686ddb12789341e4b93aa62ef5d35a9d P 2a41c90fd655498f939d826eacb0be02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:25.120752 5505 tablet_replica.cc:330] T 686ddb12789341e4b93aa62ef5d35a9d P 0bcbfe6c8cf5486dabdbe0b92c6c1264: stopping tablet replica
I20260619 08:56:25.120855 5505 raft_consensus.cc:2229] T 686ddb12789341e4b93aa62ef5d35a9d P 0bcbfe6c8cf5486dabdbe0b92c6c1264 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:56:25.120954 5505 raft_consensus.cc:2258] T 686ddb12789341e4b93aa62ef5d35a9d P 0bcbfe6c8cf5486dabdbe0b92c6c1264 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:25.121306 5505 ts_tablet_manager.cc:1905] T 686ddb12789341e4b93aa62ef5d35a9d P 0bcbfe6c8cf5486dabdbe0b92c6c1264: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:25.122831 5505 ts_tablet_manager.cc:1918] T 686ddb12789341e4b93aa62ef5d35a9d P 0bcbfe6c8cf5486dabdbe0b92c6c1264: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:56:25.122907 5505 log.cc:1196] T 686ddb12789341e4b93aa62ef5d35a9d P 0bcbfe6c8cf5486dabdbe0b92c6c1264: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-1/wal/wals/686ddb12789341e4b93aa62ef5d35a9d
I20260619 08:56:25.123123 5505 ts_tablet_manager.cc:1939] T 686ddb12789341e4b93aa62ef5d35a9d P 0bcbfe6c8cf5486dabdbe0b92c6c1264: Deleting consensus metadata
I20260619 08:56:25.123507 5003 catalog_manager.cc:4841] TS 0bcbfe6c8cf5486dabdbe0b92c6c1264 (127.0.98.2:33929): tablet 686ddb12789341e4b93aa62ef5d35a9d (table test-table [id=f7dceb86d5dd4398b55b51264841ed12]) successfully deleted
I20260619 08:56:25.123734 5504 ts_tablet_manager.cc:1905] T 686ddb12789341e4b93aa62ef5d35a9d P 2a41c90fd655498f939d826eacb0be02: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 08:56:25.124526 5002 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42463 (error 108)
I20260619 08:56:25.124812 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 5207
I20260619 08:56:25.125425 5504 ts_tablet_manager.cc:1918] T 686ddb12789341e4b93aa62ef5d35a9d P 2a41c90fd655498f939d826eacb0be02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:56:25.125492 5504 log.cc:1196] T 686ddb12789341e4b93aa62ef5d35a9d P 2a41c90fd655498f939d826eacb0be02: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859361540516-392-0/minicluster-data/ts-2/wal/wals/686ddb12789341e4b93aa62ef5d35a9d
I20260619 08:56:25.125741 5504 ts_tablet_manager.cc:1939] T 686ddb12789341e4b93aa62ef5d35a9d P 2a41c90fd655498f939d826eacb0be02: Deleting consensus metadata
I20260619 08:56:25.126017 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:25.125884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42463 (local address 127.0.0.1:34500)
0619 08:56:25.125939 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:25.125984 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 08:56:25.126173 5002 catalog_manager.cc:4584] TS eb9635c71d4d41b28881a803fc9ad0f3 (127.0.98.1:42463): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 686ddb12789341e4b93aa62ef5d35a9d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42463: connect: Connection refused (error 111)
I20260619 08:56:25.126287 5002 catalog_manager.cc:4841] TS 2a41c90fd655498f939d826eacb0be02 (127.0.98.3:38587): tablet 686ddb12789341e4b93aa62ef5d35a9d (table test-table [id=f7dceb86d5dd4398b55b51264841ed12]) successfully deleted
I20260619 08:56:25.129472 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 5338
I20260619 08:56:25.134034 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 4985
2026-06-19T08:56:25Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (887 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 08:56:25.152007 392 mini_oidc.cc:152] Starting JWKS server
I20260619 08:56:25.153038 392 webserver.cc:460] Webserver started at https://127.0.0.1:34783/ using document root <none> and password file <none>
I20260619 08:56:25.153116 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 08:56:25.153263 392 webserver.cc:460] Webserver started at http://127.0.0.1:34707/ using document root <none> and password file <none>
2026-06-19T08:56:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:56:25Z Disabled control of system clock
I20260619 08:56:25.160573 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44623
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44623
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34783/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859361540516-392-0/testchainca.cert with env {}
W20260619 08:56:25.239288 5512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:25.239497 5512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:25.239518 5512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:25.240679 5512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:56:25.240713 5512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:25.240724 5512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:25.240742 5512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:56:25.240753 5512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 08:56:25.240763 5512 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 08:56:25.240773 5512 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34783/jwks.json
I20260619 08:56:25.242226 5512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44623
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34783/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859361540516-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:25.242416 5512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:25.242570 5512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 08:56:25.247524 5512 server_base.cc:1034] running on GCE node
W20260619 08:56:25.247509 5520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:25.247498 5518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:25.247506 5517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:25.248072 5512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:25.248404 5512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:25.249547 5512 hybrid_clock.cc:648] HybridClock initialized: now 1781859385249536 us; error 30 us; skew 500 ppm
I20260619 08:56:25.250723 5512 webserver.cc:460] Webserver started at http://127.0.98.62:46585/ using document root <none> and password file <none>
I20260619 08:56:25.250974 5512 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:25.251022 5512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:25.251145 5512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:25.252249 5512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "742bdf3ac0094bb89e114035fc88cc1f"
format_stamp: "Formatted at 2026-06-19 08:56:25 on dist-test-slave-h4g5"
server_key: "7fb68b312eaf91ae277c352f01a6c1aa"
server_key_iv: "78728ad9cc1fd5cc2f6bc30dc58a2aeb"
server_key_version: "encryptionkey@0"
I20260619 08:56:25.252578 5512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "742bdf3ac0094bb89e114035fc88cc1f"
format_stamp: "Formatted at 2026-06-19 08:56:25 on dist-test-slave-h4g5"
server_key: "7fb68b312eaf91ae277c352f01a6c1aa"
server_key_iv: "78728ad9cc1fd5cc2f6bc30dc58a2aeb"
server_key_version: "encryptionkey@0"
I20260619 08:56:25.253804 5512 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:25.254562 5526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:25.254719 5512 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:25.254802 5512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "742bdf3ac0094bb89e114035fc88cc1f"
format_stamp: "Formatted at 2026-06-19 08:56:25 on dist-test-slave-h4g5"
server_key: "7fb68b312eaf91ae277c352f01a6c1aa"
server_key_iv: "78728ad9cc1fd5cc2f6bc30dc58a2aeb"
server_key_version: "encryptionkey@0"
I20260619 08:56:25.254865 5512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:25.278438 5512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:25.278765 5512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:25.278898 5512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:25.282711 5512 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44623
I20260619 08:56:25.282760 5578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44623 every 8 connection(s)
I20260619 08:56:25.283077 5512 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:56:25.283994 5579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:25.284618 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 5512
I20260619 08:56:25.284698 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 08:56:25.284870 392 external_mini_cluster.cc:1428] Setting key 559ca11b0485bb840d561f052b8ceb80
I20260619 08:56:25.286911 5579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 742bdf3ac0094bb89e114035fc88cc1f: Bootstrap starting.
I20260619 08:56:25.287588 5579 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 742bdf3ac0094bb89e114035fc88cc1f: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:25.287868 5579 log.cc:824] T 00000000000000000000000000000000 P 742bdf3ac0094bb89e114035fc88cc1f: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:25.288614 5579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 742bdf3ac0094bb89e114035fc88cc1f: No bootstrap required, opened a new log
I20260619 08:56:25.290807 5582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:25.285344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56964 (local address 127.0.98.62:44623)
0619 08:56:25.285540 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:25.285548 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:25.285581 (+ 33us) server_negotiation.cc:407] Connection header received
0619 08:56:25.286087 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:25.286096 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:25.286181 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:25.286328 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:25.286483 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:25.287773 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:25.288726 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:25.289337 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:25.289462 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:25.290109 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:25.290113 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:25.290113 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:56:25.290317 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:25.290361 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:56:25.290377 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:25.290448 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:25.290506 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 08:56:25.290590 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
I20260619 08:56:25.290964 5579 raft_consensus.cc:348] T 00000000000000000000000000000000 P 742bdf3ac0094bb89e114035fc88cc1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742bdf3ac0094bb89e114035fc88cc1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44623 } }
I20260619 08:56:25.291096 5579 raft_consensus.cc:374] T 00000000000000000000000000000000 P 742bdf3ac0094bb89e114035fc88cc1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:25.291124 5579 raft_consensus.cc:729] T 00000000000000000000000000000000 P 742bdf3ac0094bb89e114035fc88cc1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 742bdf3ac0094bb89e114035fc88cc1f, State: Initialized, Role: FOLLOWER
I20260619 08:56:25.291250 5579 consensus_queue.cc:260] T 00000000000000000000000000000000 P 742bdf3ac0094bb89e114035fc88cc1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742bdf3ac0094bb89e114035fc88cc1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44623 } }
I20260619 08:56:25.291328 5579 raft_consensus.cc:388] T 00000000000000000000000000000000 P 742bdf3ac0094bb89e114035fc88cc1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:56:25.291405 5579 raft_consensus.cc:482] T 00000000000000000000000000000000 P 742bdf3ac0094bb89e114035fc88cc1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:56:25.291456 5579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 742bdf3ac0094bb89e114035fc88cc1f [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:25.291707 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 08:56:25.292073 5579 raft_consensus.cc:504] T 00000000000000000000000000000000 P 742bdf3ac0094bb89e114035fc88cc1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742bdf3ac0094bb89e114035fc88cc1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44623 } }
I20260619 08:56:25.292207 5579 leader_election.cc:302] T 00000000000000000000000000000000 P 742bdf3ac0094bb89e114035fc88cc1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 742bdf3ac0094bb89e114035fc88cc1f; no voters:
I20260619 08:56:25.292362 5579 leader_election.cc:288] T 00000000000000000000000000000000 P 742bdf3ac0094bb89e114035fc88cc1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:56:25.292414 5584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 742bdf3ac0094bb89e114035fc88cc1f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:25.292649 5579 sys_catalog.cc:564] T 00000000000000000000000000000000 P 742bdf3ac0094bb89e114035fc88cc1f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:56:25.292686 5584 raft_consensus.cc:686] T 00000000000000000000000000000000 P 742bdf3ac0094bb89e114035fc88cc1f [term 1 LEADER]: Becoming Leader. State: Replica: 742bdf3ac0094bb89e114035fc88cc1f, State: Running, Role: LEADER
I20260619 08:56:25.292810 5584 consensus_queue.cc:237] T 00000000000000000000000000000000 P 742bdf3ac0094bb89e114035fc88cc1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742bdf3ac0094bb89e114035fc88cc1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44623 } }
I20260619 08:56:25.293143 5586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 742bdf3ac0094bb89e114035fc88cc1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 742bdf3ac0094bb89e114035fc88cc1f. Latest consensus state: current_term: 1 leader_uuid: "742bdf3ac0094bb89e114035fc88cc1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742bdf3ac0094bb89e114035fc88cc1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44623 } } }
I20260619 08:56:25.293167 5585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 742bdf3ac0094bb89e114035fc88cc1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "742bdf3ac0094bb89e114035fc88cc1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742bdf3ac0094bb89e114035fc88cc1f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44623 } } }
I20260619 08:56:25.293298 5586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 742bdf3ac0094bb89e114035fc88cc1f [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:25.293345 5585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 742bdf3ac0094bb89e114035fc88cc1f [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:25.293712 5589 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:56:25.294144 5589 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:56:25.295595 5589 catalog_manager.cc:1347] Generated new cluster ID: 1df3425bd7644364a8f567132c7d68fa
I20260619 08:56:25.295647 5589 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:56:25.306608 5589 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:56:25.307103 5589 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:56:25.322832 5589 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 742bdf3ac0094bb89e114035fc88cc1f: Generated new TSK 0
I20260619 08:56:25.323065 5589 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 08:56:25.353170 5609 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 08:56:25.353266 5609 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 08:56:25.353513 5582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:25.337887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56974 (local address 127.0.98.62:44623)
0619 08:56:25.337937 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:25.337939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:25.338030 (+ 91us) server_negotiation.cc:407] Connection header received
0619 08:56:25.338121 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:25.338122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:25.338133 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:25.338222 (+ 89us) server_negotiation.cc:226] Negotiated authn=JWT
0619 08:56:25.338340 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:25.339118 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:25.339926 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:25.340274 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:25.340345 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:25.340531 (+ 186us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 08:56:25.353286 (+ 12755us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 08:56:25.353441 (+ 155us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56974: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":26}
W20260619 08:56:25.353677 5582 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56974: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 08:56:25.353897 5608 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:25.337808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44623 (local address 127.0.0.1:56974)
0619 08:56:25.337967 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:25.337976 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:25.338033 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:25.338225 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:25.338226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:25.338230 (+ 4us) client_negotiation.cc:189] Negotiated authn=JWT
0619 08:56:25.338282 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:25.338284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:25.339228 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:25.339229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:25.339851 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:25.339853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:25.340344 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:25.340348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:25.340399 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:25.340418 (+ 19us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 08:56:25.353458 (+ 13040us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 08:56:25.353609 (+ 151us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:44623: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:44623: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34783/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 08:56:25.354282 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 5512
2026-06-19T08:56:25Z chronyd exiting
I20260619 08:56:25.389686 392 test_util.cc:175] -----------------------------------------------
I20260619 08:56:25.389770 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859361540516-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (238 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 08:56:25.390514 392 mini_oidc.cc:152] Starting JWKS server
I20260619 08:56:25.391163 392 webserver.cc:460] Webserver started at https://127.0.0.1:45109/ using document root <none> and password file <none>
I20260619 08:56:25.391211 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 08:56:25.391341 392 webserver.cc:460] Webserver started at http://127.0.0.1:38303/ using document root <none> and password file <none>
2026-06-19T08:56:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:56:25Z Disabled control of system clock
I20260619 08:56:25.398126 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34879
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34879
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45109/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859361540516-392-0/testchainca.cert with env {}
W20260619 08:56:25.475198 5615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:25.475428 5615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:25.475458 5615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:25.476696 5615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:56:25.476743 5615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:25.476760 5615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:25.476776 5615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:56:25.476791 5615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 08:56:25.476809 5615 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 08:56:25.476826 5615 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45109/jwks.json
I20260619 08:56:25.478406 5615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34879
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45109/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859361540516-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:25.478655 5615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:25.478852 5615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:25.483477 5623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:25.483521 5620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:25.483479 5621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:25.483724 5615 server_base.cc:1034] running on GCE node
I20260619 08:56:25.483971 5615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:25.484273 5615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:25.485442 5615 hybrid_clock.cc:648] HybridClock initialized: now 1781859385485427 us; error 34 us; skew 500 ppm
I20260619 08:56:25.486667 5615 webserver.cc:460] Webserver started at http://127.0.98.62:38613/ using document root <none> and password file <none>
I20260619 08:56:25.486892 5615 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:25.486941 5615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:25.487059 5615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:25.488127 5615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "d20271bf37d044d9b431ce422f9b0673"
format_stamp: "Formatted at 2026-06-19 08:56:25 on dist-test-slave-h4g5"
server_key: "2691b4daed0fe8620f43ca3db69282c7"
server_key_iv: "3eb72bd0d507b2c629980c9d87cc770c"
server_key_version: "encryptionkey@0"
I20260619 08:56:25.488466 5615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "d20271bf37d044d9b431ce422f9b0673"
format_stamp: "Formatted at 2026-06-19 08:56:25 on dist-test-slave-h4g5"
server_key: "2691b4daed0fe8620f43ca3db69282c7"
server_key_iv: "3eb72bd0d507b2c629980c9d87cc770c"
server_key_version: "encryptionkey@0"
I20260619 08:56:25.489645 5615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:25.490316 5629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:25.490474 5615 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:25.490554 5615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "d20271bf37d044d9b431ce422f9b0673"
format_stamp: "Formatted at 2026-06-19 08:56:25 on dist-test-slave-h4g5"
server_key: "2691b4daed0fe8620f43ca3db69282c7"
server_key_iv: "3eb72bd0d507b2c629980c9d87cc770c"
server_key_version: "encryptionkey@0"
I20260619 08:56:25.490619 5615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:25.504833 5615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:25.505112 5615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:25.505239 5615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:25.508827 5615 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34879
I20260619 08:56:25.508876 5681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34879 every 8 connection(s)
I20260619 08:56:25.509189 5615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:56:25.510182 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:25.511910 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 5615
I20260619 08:56:25.511997 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 08:56:25.512187 392 external_mini_cluster.cc:1428] Setting key 0cbb9ef0c725c2482569e0179cb8a8ed
I20260619 08:56:25.512745 5682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d20271bf37d044d9b431ce422f9b0673: Bootstrap starting.
I20260619 08:56:25.513505 5682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d20271bf37d044d9b431ce422f9b0673: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:25.513841 5682 log.cc:824] T 00000000000000000000000000000000 P d20271bf37d044d9b431ce422f9b0673: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:25.514518 5682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d20271bf37d044d9b431ce422f9b0673: No bootstrap required, opened a new log
I20260619 08:56:25.516359 5682 raft_consensus.cc:348] T 00000000000000000000000000000000 P d20271bf37d044d9b431ce422f9b0673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d20271bf37d044d9b431ce422f9b0673" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } }
I20260619 08:56:25.516505 5682 raft_consensus.cc:374] T 00000000000000000000000000000000 P d20271bf37d044d9b431ce422f9b0673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:25.516553 5682 raft_consensus.cc:729] T 00000000000000000000000000000000 P d20271bf37d044d9b431ce422f9b0673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d20271bf37d044d9b431ce422f9b0673, State: Initialized, Role: FOLLOWER
I20260619 08:56:25.516680 5682 consensus_queue.cc:260] T 00000000000000000000000000000000 P d20271bf37d044d9b431ce422f9b0673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d20271bf37d044d9b431ce422f9b0673" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } }
I20260619 08:56:25.516767 5682 raft_consensus.cc:388] T 00000000000000000000000000000000 P d20271bf37d044d9b431ce422f9b0673 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:56:25.516808 5682 raft_consensus.cc:482] T 00000000000000000000000000000000 P d20271bf37d044d9b431ce422f9b0673 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:56:25.516860 5682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d20271bf37d044d9b431ce422f9b0673 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:25.517422 5682 raft_consensus.cc:504] T 00000000000000000000000000000000 P d20271bf37d044d9b431ce422f9b0673 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d20271bf37d044d9b431ce422f9b0673" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } }
I20260619 08:56:25.517536 5682 leader_election.cc:302] T 00000000000000000000000000000000 P d20271bf37d044d9b431ce422f9b0673 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d20271bf37d044d9b431ce422f9b0673; no voters:
I20260619 08:56:25.517544 5685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:25.512663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41710 (local address 127.0.98.62:34879)
0619 08:56:25.512887 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:25.512896 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:25.512930 (+ 34us) server_negotiation.cc:407] Connection header received
0619 08:56:25.513494 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:25.513503 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:25.513587 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:25.513766 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:25.514037 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:25.514956 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:25.515722 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:25.516178 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:25.516290 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:25.516976 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:25.516980 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:25.516980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:56:25.517140 (+ 160us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:25.517172 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:56:25.517181 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:25.517233 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:25.517305 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 08:56:25.517383 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":91,"threads_started":1}
I20260619 08:56:25.517716 5682 leader_election.cc:288] T 00000000000000000000000000000000 P d20271bf37d044d9b431ce422f9b0673 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:56:25.517748 5687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d20271bf37d044d9b431ce422f9b0673 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:25.517920 5687 raft_consensus.cc:686] T 00000000000000000000000000000000 P d20271bf37d044d9b431ce422f9b0673 [term 1 LEADER]: Becoming Leader. State: Replica: d20271bf37d044d9b431ce422f9b0673, State: Running, Role: LEADER
I20260619 08:56:25.517966 5682 sys_catalog.cc:564] T 00000000000000000000000000000000 P d20271bf37d044d9b431ce422f9b0673 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:56:25.518034 5687 consensus_queue.cc:237] T 00000000000000000000000000000000 P d20271bf37d044d9b431ce422f9b0673 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d20271bf37d044d9b431ce422f9b0673" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } }
I20260619 08:56:25.518370 5687 sys_catalog.cc:455] T 00000000000000000000000000000000 P d20271bf37d044d9b431ce422f9b0673 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d20271bf37d044d9b431ce422f9b0673. Latest consensus state: current_term: 1 leader_uuid: "d20271bf37d044d9b431ce422f9b0673" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d20271bf37d044d9b431ce422f9b0673" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } } }
I20260619 08:56:25.518421 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 08:56:25.518379 5689 sys_catalog.cc:455] T 00000000000000000000000000000000 P d20271bf37d044d9b431ce422f9b0673 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d20271bf37d044d9b431ce422f9b0673" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d20271bf37d044d9b431ce422f9b0673" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } } }
I20260619 08:56:25.518491 5687 sys_catalog.cc:458] T 00000000000000000000000000000000 P d20271bf37d044d9b431ce422f9b0673 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:25.518540 5689 sys_catalog.cc:458] T 00000000000000000000000000000000 P d20271bf37d044d9b431ce422f9b0673 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:25.519299 5694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:56:25.519827 5694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:56:25.521253 5694 catalog_manager.cc:1347] Generated new cluster ID: eab6fc6bfdee4188bfa2d4e7c9122f1f
I20260619 08:56:25.521306 5694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:56:25.527886 5694 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:56:25.528631 5694 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:56:25.531220 5694 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d20271bf37d044d9b431ce422f9b0673: Generated new TSK 0
I20260619 08:56:25.531387 5694 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 08:56:25.553205 5712 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 08:56:25.553298 5712 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 08:56:25.553638 5685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:25.539731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41722 (local address 127.0.98.62:34879)
0619 08:56:25.539785 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:25.539788 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:25.539837 (+ 49us) server_negotiation.cc:407] Connection header received
0619 08:56:25.539919 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:25.539920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:25.539931 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:25.539991 (+ 60us) server_negotiation.cc:226] Negotiated authn=JWT
0619 08:56:25.540138 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:25.540632 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:25.541408 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:25.541607 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:25.541660 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:25.541820 (+ 160us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 08:56:25.553419 (+ 11599us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45109/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 08:56:25.553570 (+ 151us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41722: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45109/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":31}
W20260619 08:56:25.553701 5685 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41722: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45109/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 08:56:25.553725 5711 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:25.539660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34879 (local address 127.0.0.1:41722)
0619 08:56:25.539776 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:25.539794 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:25.539852 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:25.539995 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:25.539996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:25.539997 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0619 08:56:25.540052 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:25.540054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:25.540720 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:25.540721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:25.541331 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:25.541333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:25.541664 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:25.541668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:25.541727 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:25.541732 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 08:56:25.553536 (+ 11804us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45109/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 08:56:25.553637 (+ 101us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34879: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45109/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":71,"threads_started":1}
I20260619 08:56:25.553993 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 5615
2026-06-19T08:56:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (199 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 08:56:25.590292 392 mini_oidc.cc:152] Starting JWKS server
I20260619 08:56:25.590978 392 webserver.cc:460] Webserver started at https://127.0.0.1:46765/ using document root <none> and password file <none>
I20260619 08:56:25.591019 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 08:56:25.591149 392 webserver.cc:460] Webserver started at http://127.0.0.1:46543/ using document root <none> and password file <none>
2026-06-19T08:56:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:56:25Z Disabled control of system clock
I20260619 08:56:25.598019 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42191
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42191
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46765/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 08:56:25.675689 5718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:25.675897 5718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:25.675926 5718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:25.677105 5718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:56:25.677146 5718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:25.677165 5718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:25.677181 5718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:56:25.677196 5718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 08:56:25.677218 5718 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 08:56:25.677237 5718 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46765/jwks.json
I20260619 08:56:25.678633 5718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42191
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46765/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:25.678870 5718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:25.679157 5718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:25.683858 5726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:25.683995 5718 server_base.cc:1034] running on GCE node
W20260619 08:56:25.683866 5724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:25.683866 5723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:25.684337 5718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:25.684603 5718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:25.685771 5718 hybrid_clock.cc:648] HybridClock initialized: now 1781859385685753 us; error 34 us; skew 500 ppm
I20260619 08:56:25.686988 5718 webserver.cc:460] Webserver started at http://127.0.98.62:37427/ using document root <none> and password file <none>
I20260619 08:56:25.687215 5718 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:25.687264 5718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:25.687387 5718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:25.688505 5718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "0028ca3767874056b3492d159cad0c21"
format_stamp: "Formatted at 2026-06-19 08:56:25 on dist-test-slave-h4g5"
server_key: "ce95a691018cd1c3f1276b9f53522878"
server_key_iv: "cd9f5ec593dac934506be8c1b0cc6be8"
server_key_version: "encryptionkey@0"
I20260619 08:56:25.688840 5718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "0028ca3767874056b3492d159cad0c21"
format_stamp: "Formatted at 2026-06-19 08:56:25 on dist-test-slave-h4g5"
server_key: "ce95a691018cd1c3f1276b9f53522878"
server_key_iv: "cd9f5ec593dac934506be8c1b0cc6be8"
server_key_version: "encryptionkey@0"
I20260619 08:56:25.690024 5718 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 08:56:25.690655 5732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:25.690801 5718 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 08:56:25.690879 5718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "0028ca3767874056b3492d159cad0c21"
format_stamp: "Formatted at 2026-06-19 08:56:25 on dist-test-slave-h4g5"
server_key: "ce95a691018cd1c3f1276b9f53522878"
server_key_iv: "cd9f5ec593dac934506be8c1b0cc6be8"
server_key_version: "encryptionkey@0"
I20260619 08:56:25.690933 5718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:25.709648 5718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:25.709936 5718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:25.710043 5718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:25.713784 5718 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42191
I20260619 08:56:25.713829 5784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42191 every 8 connection(s)
I20260619 08:56:25.714164 5718 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:56:25.714885 5785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:25.717576 5785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0028ca3767874056b3492d159cad0c21: Bootstrap starting.
I20260619 08:56:25.718137 5785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0028ca3767874056b3492d159cad0c21: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:25.718392 5785 log.cc:824] T 00000000000000000000000000000000 P 0028ca3767874056b3492d159cad0c21: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:25.719087 5785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0028ca3767874056b3492d159cad0c21: No bootstrap required, opened a new log
I20260619 08:56:25.720739 5785 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0028ca3767874056b3492d159cad0c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0028ca3767874056b3492d159cad0c21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42191 } }
I20260619 08:56:25.720880 5785 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0028ca3767874056b3492d159cad0c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:25.720901 5785 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0028ca3767874056b3492d159cad0c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0028ca3767874056b3492d159cad0c21, State: Initialized, Role: FOLLOWER
I20260619 08:56:25.721004 5785 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0028ca3767874056b3492d159cad0c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0028ca3767874056b3492d159cad0c21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42191 } }
I20260619 08:56:25.721052 5785 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0028ca3767874056b3492d159cad0c21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:56:25.721076 5785 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0028ca3767874056b3492d159cad0c21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:56:25.721105 5785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0028ca3767874056b3492d159cad0c21 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:25.721578 5785 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0028ca3767874056b3492d159cad0c21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0028ca3767874056b3492d159cad0c21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42191 } }
I20260619 08:56:25.721660 5785 leader_election.cc:302] T 00000000000000000000000000000000 P 0028ca3767874056b3492d159cad0c21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0028ca3767874056b3492d159cad0c21; no voters:
I20260619 08:56:25.721796 5785 leader_election.cc:288] T 00000000000000000000000000000000 P 0028ca3767874056b3492d159cad0c21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:56:25.721876 5788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0028ca3767874056b3492d159cad0c21 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:25.722018 5785 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0028ca3767874056b3492d159cad0c21 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:56:25.722038 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 5718
I20260619 08:56:25.722078 5788 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0028ca3767874056b3492d159cad0c21 [term 1 LEADER]: Becoming Leader. State: Replica: 0028ca3767874056b3492d159cad0c21, State: Running, Role: LEADER
I20260619 08:56:25.722112 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 08:56:25.722177 5788 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0028ca3767874056b3492d159cad0c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0028ca3767874056b3492d159cad0c21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42191 } }
I20260619 08:56:25.722285 392 external_mini_cluster.cc:1428] Setting key e4bf8cbb2ba6fbe9db0d41b579780252
I20260619 08:56:25.722657 5789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0028ca3767874056b3492d159cad0c21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0028ca3767874056b3492d159cad0c21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0028ca3767874056b3492d159cad0c21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42191 } } }
I20260619 08:56:25.722734 5789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0028ca3767874056b3492d159cad0c21 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:25.722959 5790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0028ca3767874056b3492d159cad0c21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0028ca3767874056b3492d159cad0c21. Latest consensus state: current_term: 1 leader_uuid: "0028ca3767874056b3492d159cad0c21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0028ca3767874056b3492d159cad0c21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42191 } } }
I20260619 08:56:25.723068 5790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0028ca3767874056b3492d159cad0c21 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:25.723147 5798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:56:25.723712 5798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:56:25.725315 5798 catalog_manager.cc:1347] Generated new cluster ID: b2b9fbaa52a04c70ac4a0f4322e19b97
I20260619 08:56:25.725373 5798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:56:25.728461 5800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:25.722920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38862 (local address 127.0.98.62:42191)
0619 08:56:25.723132 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:25.723141 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:25.723175 (+ 34us) server_negotiation.cc:407] Connection header received
0619 08:56:25.723296 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:25.723342 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:25.723422 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:25.723569 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:25.723746 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:25.725069 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:25.726102 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:25.726752 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:25.726920 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:25.727689 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:25.727692 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:25.727693 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:56:25.727927 (+ 234us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:25.727966 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:56:25.727980 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:25.728032 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:25.728170 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 08:56:25.728244 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":70,"threads_started":1}
I20260619 08:56:25.729409 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 08:56:25.739099 5798 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:56:25.739806 5798 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:56:25.744724 5798 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0028ca3767874056b3492d159cad0c21: Generated new TSK 0
I20260619 08:56:25.744884 5798 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:56:25.755596 5800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:25.751958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38876 (local address 127.0.98.62:42191)
0619 08:56:25.752015 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:25.752017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:25.752169 (+ 152us) server_negotiation.cc:407] Connection header received
0619 08:56:25.752230 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:25.752230 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:25.752258 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:25.752329 (+ 71us) server_negotiation.cc:226] Negotiated authn=JWT
0619 08:56:25.752447 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:25.752935 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:25.754068 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:25.754285 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:25.754348 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:25.754550 (+ 202us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 08:56:25.755396 (+ 846us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46765/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 08:56:25.755522 (+ 126us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38876: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46765/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":32}
W20260619 08:56:25.755662 5800 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38876: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46765/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 08:56:25.755690 5814 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:25.751895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42191 (local address 127.0.0.1:38876)
0619 08:56:25.752061 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:25.752072 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:25.752168 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:25.752331 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:25.752332 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:25.752334 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0619 08:56:25.752390 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:25.752392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:25.753004 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:25.753005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:25.753971 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:25.753974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:25.754360 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:25.754367 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:25.754426 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:25.754433 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 08:56:25.755500 (+ 1067us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46765/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 08:56:25.755589 (+ 89us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42191: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46765/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":106,"threads_started":1}
I20260619 08:56:25.755945 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 5718
E20260619 08:56:25.766516 5815 webserver.cc:533] Webserver: sslize failed:
E20260619 08:56:25.766595 5815 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-19T08:56:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (202 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T08:56:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:56:25Z Disabled control of system clock
I20260619 08:56:25.799414 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40185
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40185
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859361540516-392-0/testchainca.cert with env {}
W20260619 08:56:25.876645 5819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:25.876857 5819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:25.876879 5819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:25.878063 5819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:56:25.878108 5819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:25.878119 5819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:25.878130 5819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:56:25.878140 5819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 08:56:25.878151 5819 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 08:56:25.878163 5819 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 08:56:25.879669 5819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40185
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859361540516-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:25.879909 5819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:25.880132 5819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:25.884791 5824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:25.884799 5825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:25.884804 5827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:25.884881 5819 server_base.cc:1034] running on GCE node
I20260619 08:56:25.885211 5819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:25.885509 5819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:25.886688 5819 hybrid_clock.cc:648] HybridClock initialized: now 1781859385886671 us; error 33 us; skew 500 ppm
I20260619 08:56:25.887910 5819 webserver.cc:460] Webserver started at http://127.0.98.62:36399/ using document root <none> and password file <none>
I20260619 08:56:25.888164 5819 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:25.888211 5819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:25.888329 5819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:25.889384 5819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "38d9550953c44f069dfa3bdec3fad640"
format_stamp: "Formatted at 2026-06-19 08:56:25 on dist-test-slave-h4g5"
server_key: "2375a7858912a4a9d62f901f376d26e7"
server_key_iv: "69008ce50d34bdd07b0534515409952e"
server_key_version: "encryptionkey@0"
I20260619 08:56:25.889695 5819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "38d9550953c44f069dfa3bdec3fad640"
format_stamp: "Formatted at 2026-06-19 08:56:25 on dist-test-slave-h4g5"
server_key: "2375a7858912a4a9d62f901f376d26e7"
server_key_iv: "69008ce50d34bdd07b0534515409952e"
server_key_version: "encryptionkey@0"
I20260619 08:56:25.890868 5819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:56:25.891501 5833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:25.891654 5819 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:25.891732 5819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "38d9550953c44f069dfa3bdec3fad640"
format_stamp: "Formatted at 2026-06-19 08:56:25 on dist-test-slave-h4g5"
server_key: "2375a7858912a4a9d62f901f376d26e7"
server_key_iv: "69008ce50d34bdd07b0534515409952e"
server_key_version: "encryptionkey@0"
I20260619 08:56:25.891791 5819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:25.910221 5819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:25.910550 5819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:25.910676 5819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:25.914219 5819 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40185
I20260619 08:56:25.914268 5885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40185 every 8 connection(s)
I20260619 08:56:25.914584 5819 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:56:25.915563 5886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:25.917850 5886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38d9550953c44f069dfa3bdec3fad640: Bootstrap starting.
I20260619 08:56:25.918394 5886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38d9550953c44f069dfa3bdec3fad640: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:25.918645 5886 log.cc:824] T 00000000000000000000000000000000 P 38d9550953c44f069dfa3bdec3fad640: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:25.919399 5886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38d9550953c44f069dfa3bdec3fad640: No bootstrap required, opened a new log
I20260619 08:56:25.921030 5886 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38d9550953c44f069dfa3bdec3fad640 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38d9550953c44f069dfa3bdec3fad640" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40185 } }
I20260619 08:56:25.921182 5886 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38d9550953c44f069dfa3bdec3fad640 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:25.921206 5886 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38d9550953c44f069dfa3bdec3fad640 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38d9550953c44f069dfa3bdec3fad640, State: Initialized, Role: FOLLOWER
I20260619 08:56:25.921340 5886 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38d9550953c44f069dfa3bdec3fad640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38d9550953c44f069dfa3bdec3fad640" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40185 } }
I20260619 08:56:25.921403 5886 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38d9550953c44f069dfa3bdec3fad640 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:56:25.921423 5886 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38d9550953c44f069dfa3bdec3fad640 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:56:25.921453 5886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38d9550953c44f069dfa3bdec3fad640 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:25.921953 5886 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38d9550953c44f069dfa3bdec3fad640 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38d9550953c44f069dfa3bdec3fad640" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40185 } }
I20260619 08:56:25.922040 5886 leader_election.cc:302] T 00000000000000000000000000000000 P 38d9550953c44f069dfa3bdec3fad640 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38d9550953c44f069dfa3bdec3fad640; no voters:
I20260619 08:56:25.922230 5886 leader_election.cc:288] T 00000000000000000000000000000000 P 38d9550953c44f069dfa3bdec3fad640 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:56:25.922267 5889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38d9550953c44f069dfa3bdec3fad640 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:25.922427 5889 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38d9550953c44f069dfa3bdec3fad640 [term 1 LEADER]: Becoming Leader. State: Replica: 38d9550953c44f069dfa3bdec3fad640, State: Running, Role: LEADER
I20260619 08:56:25.922573 5886 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38d9550953c44f069dfa3bdec3fad640 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:56:25.922549 5889 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38d9550953c44f069dfa3bdec3fad640 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38d9550953c44f069dfa3bdec3fad640" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40185 } }
I20260619 08:56:25.922907 5890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38d9550953c44f069dfa3bdec3fad640 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38d9550953c44f069dfa3bdec3fad640. Latest consensus state: current_term: 1 leader_uuid: "38d9550953c44f069dfa3bdec3fad640" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38d9550953c44f069dfa3bdec3fad640" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40185 } } }
I20260619 08:56:25.923015 5891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38d9550953c44f069dfa3bdec3fad640 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38d9550953c44f069dfa3bdec3fad640" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38d9550953c44f069dfa3bdec3fad640" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40185 } } }
I20260619 08:56:25.923046 5890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38d9550953c44f069dfa3bdec3fad640 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:25.923416 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 5819
I20260619 08:56:25.923514 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 08:56:25.923522 5891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38d9550953c44f069dfa3bdec3fad640 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:25.923606 5898 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:56:25.923702 392 external_mini_cluster.cc:1428] Setting key 095f8dafa3388e83fc05ba351d470ccd
I20260619 08:56:25.924140 5898 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:56:25.925740 5898 catalog_manager.cc:1347] Generated new cluster ID: 76ba4e5c671247aa93f431908760bd97
I20260619 08:56:25.925791 5898 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:56:25.930199 5898 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:56:25.930845 5898 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:56:25.935355 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:25.924364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34464 (local address 127.0.98.62:40185)
0619 08:56:25.929773 (+ 5409us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:25.929780 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:25.929809 (+ 29us) server_negotiation.cc:407] Connection header received
0619 08:56:25.929930 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:25.929938 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:25.930015 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:25.930183 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:25.930421 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:25.932130 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:25.933258 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:25.933845 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:25.933956 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:25.934644 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:25.934648 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:25.934648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:56:25.934856 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:25.934909 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:56:25.934923 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:25.934979 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:25.935048 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 08:56:25.935134 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5340,"thread_start_us":74,"threads_started":1}
I20260619 08:56:25.936430 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 08:56:25.941175 5898 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38d9550953c44f069dfa3bdec3fad640: Generated new TSK 0
I20260619 08:56:25.941349 5898 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:56:26.377461 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:25.952046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34468 (local address 127.0.98.62:40185)
0619 08:56:25.952116 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:25.952118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:25.952206 (+ 88us) server_negotiation.cc:407] Connection header received
0619 08:56:25.952296 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:25.952297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:25.952309 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:25.952370 (+ 61us) server_negotiation.cc:226] Negotiated authn=JWT
0619 08:56:25.952492 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:25.952999 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:25.953760 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:25.953976 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:25.954029 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:25.954201 (+ 172us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 08:56:26.377213 (+423012us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 08:56:26.377397 (+ 184us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34468: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":22}
W20260619 08:56:26.377521 5908 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34468: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 08:56:26.377596 5915 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:25.951970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40185 (local address 127.0.0.1:34468)
0619 08:56:25.952147 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:25.952156 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:25.952212 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:25.952374 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:25.952375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:25.952377 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0619 08:56:25.952436 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:25.952439 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:25.953074 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:25.953075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:25.953678 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:25.953679 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:25.954042 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:25.954050 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:25.954107 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:25.954113 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 08:56:26.377358 (+423245us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 08:56:26.377477 (+ 119us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40185: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":80,"threads_started":1}
I20260619 08:56:26.377872 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 5819
2026-06-19T08:56:26Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (602 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (85 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (85 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:56:26 dist-test-slave-h4g5 krb5kdc[5927](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:56:26 dist-test-slave-h4g5 krb5kdc[5927](info): set up 2 sockets
Jun 19 08:56:26 dist-test-slave-h4g5 krb5kdc[5927](info): commencing operation
krb5kdc: starting...
W20260619 08:56:27.492506 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 08:56:27 dist-test-slave-h4g5 krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:56:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:56:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 08:56:27.607087 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34669
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34669
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:27.685799 5940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:27.686008 5940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:27.686048 5940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:27.687213 5940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:56:27.687253 5940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:27.687271 5940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:27.687288 5940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:56:27.687304 5940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:56:27.688853 5940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:27.689121 5940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:27.689334 5940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:27.694166 5946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:27.694159 5948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:27.694229 5945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:27.694358 5940 server_base.cc:1034] running on GCE node
I20260619 08:56:27.694573 5940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:27.694837 5940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:27.695991 5940 hybrid_clock.cc:648] HybridClock initialized: now 1781859387695960 us; error 47 us; skew 500 ppm
Jun 19 08:56:27 dist-test-slave-h4g5 krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:27.702585 5940 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 08:56:27.703040 5940 webserver.cc:460] Webserver started at http://127.0.98.62:41621/ using document root <none> and password file <none>
I20260619 08:56:27.703244 5940 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:27.703289 5940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:27.703408 5940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:27.704563 5940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "b13688f0f7ec4006af88ecbe80e0cfe1"
format_stamp: "Formatted at 2026-06-19 08:56:27 on dist-test-slave-h4g5"
server_key: "fdb26883067179f32ed87fd6c5ac6c87"
server_key_iv: "139d57c39adca99a63f959034258cbc9"
server_key_version: "encryptionkey@0"
I20260619 08:56:27.704885 5940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "b13688f0f7ec4006af88ecbe80e0cfe1"
format_stamp: "Formatted at 2026-06-19 08:56:27 on dist-test-slave-h4g5"
server_key: "fdb26883067179f32ed87fd6c5ac6c87"
server_key_iv: "139d57c39adca99a63f959034258cbc9"
server_key_version: "encryptionkey@0"
I20260619 08:56:27.706116 5940 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:56:27.706825 5955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:27.706993 5940 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:27.707082 5940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "b13688f0f7ec4006af88ecbe80e0cfe1"
format_stamp: "Formatted at 2026-06-19 08:56:27 on dist-test-slave-h4g5"
server_key: "fdb26883067179f32ed87fd6c5ac6c87"
server_key_iv: "139d57c39adca99a63f959034258cbc9"
server_key_version: "encryptionkey@0"
I20260619 08:56:27.707149 5940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:27.725937 5940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:27.728269 5940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:27.728394 5940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:27.732652 5940 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34669
I20260619 08:56:27.732741 6007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34669 every 8 connection(s)
I20260619 08:56:27.733209 5940 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:56:27.733857 6008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:27.736256 6008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b13688f0f7ec4006af88ecbe80e0cfe1: Bootstrap starting.
I20260619 08:56:27.736826 6008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b13688f0f7ec4006af88ecbe80e0cfe1: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:27.737083 6008 log.cc:824] T 00000000000000000000000000000000 P b13688f0f7ec4006af88ecbe80e0cfe1: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:27.737702 6008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b13688f0f7ec4006af88ecbe80e0cfe1: No bootstrap required, opened a new log
I20260619 08:56:27.739223 6008 raft_consensus.cc:348] T 00000000000000000000000000000000 P b13688f0f7ec4006af88ecbe80e0cfe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b13688f0f7ec4006af88ecbe80e0cfe1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34669 } }
I20260619 08:56:27.739362 6008 raft_consensus.cc:374] T 00000000000000000000000000000000 P b13688f0f7ec4006af88ecbe80e0cfe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:27.739451 6008 raft_consensus.cc:729] T 00000000000000000000000000000000 P b13688f0f7ec4006af88ecbe80e0cfe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b13688f0f7ec4006af88ecbe80e0cfe1, State: Initialized, Role: FOLLOWER
I20260619 08:56:27.739588 6008 consensus_queue.cc:260] T 00000000000000000000000000000000 P b13688f0f7ec4006af88ecbe80e0cfe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b13688f0f7ec4006af88ecbe80e0cfe1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34669 } }
I20260619 08:56:27.739657 6008 raft_consensus.cc:388] T 00000000000000000000000000000000 P b13688f0f7ec4006af88ecbe80e0cfe1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:56:27.739686 6008 raft_consensus.cc:482] T 00000000000000000000000000000000 P b13688f0f7ec4006af88ecbe80e0cfe1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:56:27.739732 6008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b13688f0f7ec4006af88ecbe80e0cfe1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:27.740635 6008 raft_consensus.cc:504] T 00000000000000000000000000000000 P b13688f0f7ec4006af88ecbe80e0cfe1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b13688f0f7ec4006af88ecbe80e0cfe1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34669 } }
I20260619 08:56:27.740731 6008 leader_election.cc:302] T 00000000000000000000000000000000 P b13688f0f7ec4006af88ecbe80e0cfe1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b13688f0f7ec4006af88ecbe80e0cfe1; no voters:
I20260619 08:56:27.740887 6008 leader_election.cc:288] T 00000000000000000000000000000000 P b13688f0f7ec4006af88ecbe80e0cfe1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:56:27.740933 6011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b13688f0f7ec4006af88ecbe80e0cfe1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:27.741104 6011 raft_consensus.cc:686] T 00000000000000000000000000000000 P b13688f0f7ec4006af88ecbe80e0cfe1 [term 1 LEADER]: Becoming Leader. State: Replica: b13688f0f7ec4006af88ecbe80e0cfe1, State: Running, Role: LEADER
I20260619 08:56:27.741128 6008 sys_catalog.cc:564] T 00000000000000000000000000000000 P b13688f0f7ec4006af88ecbe80e0cfe1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:56:27.741192 6011 consensus_queue.cc:237] T 00000000000000000000000000000000 P b13688f0f7ec4006af88ecbe80e0cfe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b13688f0f7ec4006af88ecbe80e0cfe1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34669 } }
I20260619 08:56:27.741556 6012 sys_catalog.cc:455] T 00000000000000000000000000000000 P b13688f0f7ec4006af88ecbe80e0cfe1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b13688f0f7ec4006af88ecbe80e0cfe1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b13688f0f7ec4006af88ecbe80e0cfe1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34669 } } }
I20260619 08:56:27.741683 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 5940
I20260619 08:56:27.741786 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 08:56:27.741796 6012 sys_catalog.cc:458] T 00000000000000000000000000000000 P b13688f0f7ec4006af88ecbe80e0cfe1 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:27.741961 392 external_mini_cluster.cc:1428] Setting key d79842a92c5b53d904f255fcef8646ad
I20260619 08:56:27.741708 6013 sys_catalog.cc:455] T 00000000000000000000000000000000 P b13688f0f7ec4006af88ecbe80e0cfe1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b13688f0f7ec4006af88ecbe80e0cfe1. Latest consensus state: current_term: 1 leader_uuid: "b13688f0f7ec4006af88ecbe80e0cfe1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b13688f0f7ec4006af88ecbe80e0cfe1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34669 } } }
I20260619 08:56:27.742205 6013 sys_catalog.cc:458] T 00000000000000000000000000000000 P b13688f0f7ec4006af88ecbe80e0cfe1 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:27.742345 6020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:56:27.742825 6020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:56:27.744268 6020 catalog_manager.cc:1347] Generated new cluster ID: 7ff9ebfa333040e1b473de2f742d511b
I20260619 08:56:27.744334 6020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 08:56:27 dist-test-slave-h4g5 krb5kdc[5927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:27.754439 6028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:27.742441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40404 (local address 127.0.98.62:34669)
0619 08:56:27.742593 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:27.742603 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:27.742639 (+ 36us) server_negotiation.cc:407] Connection header received
0619 08:56:27.742832 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:27.742840 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:27.742901 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:27.743059 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:27.743757 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:27.745219 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:27.746357 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:27.747058 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:27.747182 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:27.750182 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:27.750187 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:27.750188 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:27.750203 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:27.752494 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:27.753022 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:27.753025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:27.753026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:27.753168 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:27.753369 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:27.753370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:27.753370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:27.753736 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:27.753917 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:27.753999 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 08:56:27.754164 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
I20260619 08:56:27.759675 6020 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:56:27.760332 6020 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:56:27.763334 6020 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b13688f0f7ec4006af88ecbe80e0cfe1: Generated new TSK 0
I20260619 08:56:27.763540 6020 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 08:56:27.790117 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34669
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:27.868171 6034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:27.868378 6034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:27.868400 6034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:27.869532 6034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:27.869567 6034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:27.869592 6034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 08:56:27.870993 6034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:27.871183 6034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:27.871366 6034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:27.876034 6039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:27.876034 6042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:27.876130 6040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:27.876129 6034 server_base.cc:1034] running on GCE node
I20260619 08:56:27.876506 6034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:27.876811 6034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:27.877952 6034 hybrid_clock.cc:648] HybridClock initialized: now 1781859387877940 us; error 31 us; skew 500 ppm
Jun 19 08:56:27 dist-test-slave-h4g5 krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:27.880353 6034 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 08:56:27.880754 6034 webserver.cc:460] Webserver started at http://127.0.98.1:46483/ using document root <none> and password file <none>
I20260619 08:56:27.880923 6034 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:27.880960 6034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:27.881048 6034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:27.882011 6034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-0/data/instance:
uuid: "bc8f121d4b18468ca6345b32b7c79beb"
format_stamp: "Formatted at 2026-06-19 08:56:27 on dist-test-slave-h4g5"
server_key: "7592e3a0808b90abd1d00d33827fc960"
server_key_iv: "5a245e48cdf48c60d05c2f35f5ccb42b"
server_key_version: "encryptionkey@0"
I20260619 08:56:27.882298 6034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bc8f121d4b18468ca6345b32b7c79beb"
format_stamp: "Formatted at 2026-06-19 08:56:27 on dist-test-slave-h4g5"
server_key: "7592e3a0808b90abd1d00d33827fc960"
server_key_iv: "5a245e48cdf48c60d05c2f35f5ccb42b"
server_key_version: "encryptionkey@0"
I20260619 08:56:27.883546 6034 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260619 08:56:27.884343 6049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:27.884527 6034 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 08:56:27.884617 6034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-0/wal
uuid: "bc8f121d4b18468ca6345b32b7c79beb"
format_stamp: "Formatted at 2026-06-19 08:56:27 on dist-test-slave-h4g5"
server_key: "7592e3a0808b90abd1d00d33827fc960"
server_key_iv: "5a245e48cdf48c60d05c2f35f5ccb42b"
server_key_version: "encryptionkey@0"
I20260619 08:56:27.884685 6034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:27.907651 6034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:27.909817 6034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:27.909929 6034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:27.910176 6034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:27.910471 6034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:27.910512 6034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:27.910542 6034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:27.910565 6034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:27.915880 6034 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36811
I20260619 08:56:27.915972 6162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36811 every 8 connection(s)
I20260619 08:56:27.916244 6034 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 08:56:27 dist-test-slave-h4g5 krb5kdc[5927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:27.924754 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 6034
I20260619 08:56:27.924854 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-0/wal/instance
I20260619 08:56:27.925029 392 external_mini_cluster.cc:1428] Setting key 5fb8c98aaaa1ba81fbfa2719a855e34a
I20260619 08:56:27.925833 6028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:27.916969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54945 (local address 127.0.98.62:34669)
0619 08:56:27.917048 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:27.917050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:27.917147 (+ 97us) server_negotiation.cc:407] Connection header received
0619 08:56:27.918066 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:27.918067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:27.918083 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:27.918138 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:27.918811 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:27.919405 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:27.920582 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:27.920784 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:27.920847 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:27.922983 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:27.922984 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:27.922985 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:27.922994 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:27.924332 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:27.924793 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:27.924794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:27.924795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:27.924864 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:27.925171 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:27.925172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:27.925173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:27.925297 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:27.925359 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:27.925669 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 08:56:27.925749 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 08:56:27.926024 6165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:27.916884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34669 (local address 127.0.98.1:54945)
0619 08:56:27.917031 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:27.917067 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:27.917958 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:27.918205 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:27.918208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:27.918497 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:27.918698 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:27.918703 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:27.919530 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:27.919531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:27.920486 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:27.920489 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:27.920881 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:27.920881 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:27.920995 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:27.921435 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:27.921442 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:27.922894 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:27.924402 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:27.924408 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:27.924410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:27.924701 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:27.924939 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:27.924940 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:27.924940 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:27.925071 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:27.925354 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:27.925361 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:27.925526 (+ 165us) client_negotiation.cc:769] Sending connection context
0619 08:56:27.925636 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 08:56:27.925808 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
I20260619 08:56:27.927209 6163 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34669
I20260619 08:56:27.927317 6163 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:27.927495 6163 heartbeater.cc:511] Master 127.0.98.62:34669 requested a full tablet report, sending...
I20260619 08:56:27.928042 5972 ts_manager.cc:194] Registered new tserver with Master: bc8f121d4b18468ca6345b32b7c79beb (127.0.98.1:36811)
I20260619 08:56:27.928706 5972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54945
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 08:56:27.948864 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34669
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:28.030606 6168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:28.030835 6168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:28.030858 6168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:28.032107 6168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:28.032159 6168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:28.032186 6168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 08:56:28.033995 6168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:28.034276 6168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:28.034518 6168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:28.039446 6176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:28.039492 6174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:28.039453 6173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:28.039698 6168 server_base.cc:1034] running on GCE node
I20260619 08:56:28.039883 6168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:28.040158 6168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:28.041293 6168 hybrid_clock.cc:648] HybridClock initialized: now 1781859388041273 us; error 35 us; skew 500 ppm
Jun 19 08:56:28 dist-test-slave-h4g5 krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:28.043748 6168 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 08:56:28.044234 6168 webserver.cc:460] Webserver started at http://127.0.98.2:44837/ using document root <none> and password file <none>
I20260619 08:56:28.044462 6168 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:28.044510 6168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:28.044643 6168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:28.045689 6168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-1/data/instance:
uuid: "3e510cbbacb94fb4a07529375d26382c"
format_stamp: "Formatted at 2026-06-19 08:56:28 on dist-test-slave-h4g5"
server_key: "cf14dd7bf7425d4eca6b288b74f14925"
server_key_iv: "c379f668e954030567053fc77c639e3c"
server_key_version: "encryptionkey@0"
I20260619 08:56:28.046011 6168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3e510cbbacb94fb4a07529375d26382c"
format_stamp: "Formatted at 2026-06-19 08:56:28 on dist-test-slave-h4g5"
server_key: "cf14dd7bf7425d4eca6b288b74f14925"
server_key_iv: "c379f668e954030567053fc77c639e3c"
server_key_version: "encryptionkey@0"
I20260619 08:56:28.047274 6168 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 08:56:28.048032 6183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:28.048197 6168 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:28.048285 6168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-1/wal
uuid: "3e510cbbacb94fb4a07529375d26382c"
format_stamp: "Formatted at 2026-06-19 08:56:28 on dist-test-slave-h4g5"
server_key: "cf14dd7bf7425d4eca6b288b74f14925"
server_key_iv: "c379f668e954030567053fc77c639e3c"
server_key_version: "encryptionkey@0"
I20260619 08:56:28.048355 6168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:28.057288 6168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:28.059309 6168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:28.059443 6168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:28.059651 6168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:28.059937 6168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:28.059968 6168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:28.059998 6168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:28.060015 6168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:28.065450 6168 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40797
I20260619 08:56:28.065490 6296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40797 every 8 connection(s)
I20260619 08:56:28.065789 6168 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 08:56:28 dist-test-slave-h4g5 krb5kdc[5927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:28.073454 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 6168
I20260619 08:56:28.073553 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-1/wal/instance
I20260619 08:56:28.073750 392 external_mini_cluster.cc:1428] Setting key e53ef751dd687764e04102a15edb630f
I20260619 08:56:28.075032 6028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:28.066434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40287 (local address 127.0.98.62:34669)
0619 08:56:28.066479 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:28.066481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:28.066678 (+ 197us) server_negotiation.cc:407] Connection header received
0619 08:56:28.067614 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:28.067615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:28.067629 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:28.067681 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:28.068333 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:28.068829 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:28.069807 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:28.069996 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:28.070051 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:28.072122 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:28.072124 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:28.072124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:28.072133 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:28.073332 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:28.073881 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:28.073882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:28.073882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:28.073948 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:28.074269 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:28.074270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:28.074270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:28.074428 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:28.074511 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:28.074834 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 08:56:28.074926 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 08:56:28.075093 6299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:28.066412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34669 (local address 127.0.98.2:40287)
0619 08:56:28.066556 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:28.066594 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:28.067494 (+ 900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:28.067751 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:28.067756 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:28.068005 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:28.068248 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:28.068253 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:28.068907 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:28.068908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:28.069726 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:28.069728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:28.070060 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:28.070061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:28.070201 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:28.070640 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:28.070648 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:28.071984 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:28.073416 (+ 1432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:28.073423 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:28.073425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:28.073768 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:28.074011 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:28.074012 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:28.074012 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:28.074169 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:28.074522 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:28.074529 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:28.074698 (+ 169us) client_negotiation.cc:769] Sending connection context
0619 08:56:28.074791 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 08:56:28.074921 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":73,"threads_started":1}
I20260619 08:56:28.075887 6297 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34669
I20260619 08:56:28.075994 6297 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:28.076213 6297 heartbeater.cc:511] Master 127.0.98.62:34669 requested a full tablet report, sending...
I20260619 08:56:28.076611 5972 ts_manager.cc:194] Registered new tserver with Master: 3e510cbbacb94fb4a07529375d26382c (127.0.98.2:40797)
I20260619 08:56:28.076972 5972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40287
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 08:56:28.098294 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34669
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:28.175602 6302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:28.175801 6302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:28.175822 6302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:28.176986 6302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:28.177031 6302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:28.177057 6302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 08:56:28.178500 6302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:28.178725 6302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:28.178931 6302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:28.183784 6307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:28.183908 6302 server_base.cc:1034] running on GCE node
W20260619 08:56:28.183853 6308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:28.183799 6310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:28.184217 6302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:28.184422 6302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:28.185580 6302 hybrid_clock.cc:648] HybridClock initialized: now 1781859388185561 us; error 37 us; skew 500 ppm
Jun 19 08:56:28 dist-test-slave-h4g5 krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:28.188026 6302 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 08:56:28.188489 6302 webserver.cc:460] Webserver started at http://127.0.98.3:40811/ using document root <none> and password file <none>
I20260619 08:56:28.188679 6302 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:28.188721 6302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:28.188836 6302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:28.189860 6302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-2/data/instance:
uuid: "68116d8a50a149ad8bfece9f404c38f0"
format_stamp: "Formatted at 2026-06-19 08:56:28 on dist-test-slave-h4g5"
server_key: "ac4ca5dc8861bec537c0395e9aa4e416"
server_key_iv: "e2f513ad5e652a0d5130d254c0b43faa"
server_key_version: "encryptionkey@0"
I20260619 08:56:28.190200 6302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-2/wal/instance:
uuid: "68116d8a50a149ad8bfece9f404c38f0"
format_stamp: "Formatted at 2026-06-19 08:56:28 on dist-test-slave-h4g5"
server_key: "ac4ca5dc8861bec537c0395e9aa4e416"
server_key_iv: "e2f513ad5e652a0d5130d254c0b43faa"
server_key_version: "encryptionkey@0"
I20260619 08:56:28.191461 6302 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 08:56:28.192215 6317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:28.192345 6302 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:28.192435 6302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-2/wal
uuid: "68116d8a50a149ad8bfece9f404c38f0"
format_stamp: "Formatted at 2026-06-19 08:56:28 on dist-test-slave-h4g5"
server_key: "ac4ca5dc8861bec537c0395e9aa4e416"
server_key_iv: "e2f513ad5e652a0d5130d254c0b43faa"
server_key_version: "encryptionkey@0"
I20260619 08:56:28.192500 6302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:28.219928 6302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:28.221776 6302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:28.221897 6302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:28.222126 6302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:28.222424 6302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:28.222456 6302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:28.222477 6302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:28.222492 6302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:28.228072 6302 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35539
I20260619 08:56:28.228148 6430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35539 every 8 connection(s)
I20260619 08:56:28.228466 6302 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
I20260619 08:56:28.232988 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 6302
I20260619 08:56:28.233084 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/minicluster-data/ts-2/wal/instance
I20260619 08:56:28.233266 392 external_mini_cluster.cc:1428] Setting key 86668ff6a24b94ef1dea1374b08ece3c
Jun 19 08:56:28 dist-test-slave-h4g5 krb5kdc[5927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:28.237308 6028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:28.229199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48513 (local address 127.0.98.62:34669)
0619 08:56:28.229250 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:28.229252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:28.229418 (+ 166us) server_negotiation.cc:407] Connection header received
0619 08:56:28.230310 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:28.230311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:28.230322 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:28.230382 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:28.230954 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:28.231434 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:28.232414 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:28.232630 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:28.232687 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:28.234881 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:28.234883 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:28.234883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:28.234890 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:28.236196 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:28.236557 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:28.236558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:28.236559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:28.236619 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:28.236817 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:28.236817 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:28.236818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:28.236916 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:28.236963 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:28.237179 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 08:56:28.237238 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 08:56:28.237475 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:28.229135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34669 (local address 127.0.98.3:48513)
0619 08:56:28.229298 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:28.229339 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:28.230222 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:28.230438 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:28.230441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:28.230690 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:28.230877 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:28.230881 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:28.231499 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:28.231500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:28.232330 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:28.232332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:28.232695 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:28.232696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:28.232795 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:28.233366 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:28.233377 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:28.234795 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:28.236266 (+ 1471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:28.236271 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:28.236273 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:28.236476 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:28.236683 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:28.236683 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:28.236684 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:28.236774 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:28.236953 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:28.236956 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:28.237065 (+ 109us) client_negotiation.cc:769] Sending connection context
0619 08:56:28.237157 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 08:56:28.237273 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":76,"threads_started":1}
I20260619 08:56:28.237871 6431 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34669
I20260619 08:56:28.237985 6431 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:28.238155 6431 heartbeater.cc:511] Master 127.0.98.62:34669 requested a full tablet report, sending...
I20260619 08:56:28.238641 5972 ts_manager.cc:194] Registered new tserver with Master: 68116d8a50a149ad8bfece9f404c38f0 (127.0.98.3:35539)
I20260619 08:56:28.239014 5972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48513
I20260619 08:56:28.245218 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/krb5kdc/test-admin.keytab.
Jun 19 08:56:28 dist-test-slave-h4g5 krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:28.260643 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 08:56:28.262156 6438 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:28.261511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43876 (local address 127.0.98.1:36811)
0619 08:56:28.261674 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:28.261679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:28.261689 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:56:28.261714 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:28.261716 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:28.261728 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:28.261794 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:28.262095 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43876: BlockingRecv error: recv got EOF from 127.0.0.1:43876 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":93,"threads_started":1}
W20260619 08:56:28.262362 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/krb5kdc/krb5cc' of type FILE
Jun 19 08:56:28 dist-test-slave-h4g5 krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:28.263302 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 08:56:28 dist-test-slave-h4g5 krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:28.264240 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 08:56:28 dist-test-slave-h4g5 krb5kdc[5927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 08:56:28.271327 6438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:28.264701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43892 (local address 127.0.98.1:36811)
0619 08:56:28.264757 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:28.264759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:28.264768 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:56:28.264837 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:28.264838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:28.264849 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:28.264896 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:28.265275 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:28.265984 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:28.266681 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:28.267035 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:28.267096 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:28.269117 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:28.269125 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:28.269126 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:28.269133 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:28.270435 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:28.270740 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:28.270743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:28.270743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:28.270793 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:28.270926 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:28.270926 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:28.270926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:28.271044 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:28.271136 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:28.271190 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 08:56:28.271251 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260619 08:56:28.272969 6438 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:28.272439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43894 (local address 127.0.98.1:36811)
0619 08:56:28.272488 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:28.272490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:28.272518 (+ 28us) server_negotiation.cc:407] Connection header received
0619 08:56:28.272598 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:28.272598 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:28.272607 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:28.272647 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:28.272918 (+ 271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43894: BlockingRecv error: recv got EOF from 127.0.0.1:43894 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260619 08:56:28.273038 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/krb5kdc/krb5cc' of type FILE
Jun 19 08:56:28 dist-test-slave-h4g5 krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:28.273909 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 08:56:28 dist-test-slave-h4g5 krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:28.274677 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 08:56:28 dist-test-slave-h4g5 krb5kdc[5927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 08:56:28.280865 6438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:28.275148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43900 (local address 127.0.98.1:36811)
0619 08:56:28.275184 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:28.275186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:28.275193 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:56:28.275231 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:28.275231 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:28.275238 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:28.275279 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:28.275570 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:28.276036 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:28.276704 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:28.276895 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:28.276942 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:28.278743 (+ 1801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:28.278744 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:28.278744 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:28.278750 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:28.279946 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:28.280278 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:28.280279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:28.280279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:28.280325 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:28.280451 (+ 126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:28.280452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:28.280452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:28.280564 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:28.280629 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:28.280730 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 08:56:28.280784 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260619 08:56:28.282251 6438 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:28.281751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43914 (local address 127.0.98.1:36811)
0619 08:56:28.281799 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:28.281801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:28.281856 (+ 55us) server_negotiation.cc:407] Connection header received
0619 08:56:28.281923 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:28.281924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:28.281933 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:28.281971 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:28.282220 (+ 249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43914: BlockingRecv error: recv got EOF from 127.0.0.1:43914 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260619 08:56:28.282404 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859361540516-392-0/krb5kdc/krb5cc' of type FILE
Jun 19 08:56:28 dist-test-slave-h4g5 krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:28.283361 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 08:56:28 dist-test-slave-h4g5 krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:28.284184 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 08:56:28 dist-test-slave-h4g5 krb5kdc[5927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 08:56:28.290423 6438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:28.284626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43926 (local address 127.0.98.1:36811)
0619 08:56:28.284686 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:28.284688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:28.284717 (+ 29us) server_negotiation.cc:407] Connection header received
0619 08:56:28.284783 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:28.284784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:28.284792 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:28.284835 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:28.285169 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:28.285632 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:28.286275 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:28.286457 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:28.286509 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:28.288437 (+ 1928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:28.288439 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:28.288439 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:28.288445 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:28.289511 (+ 1066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:28.289783 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:28.289785 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:28.289785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:28.289832 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:28.289984 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:28.289985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:28.289985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:28.290069 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:28.290144 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:28.290266 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 08:56:28.290334 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 08:56:28.291152 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 6034
I20260619 08:56:28.295356 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 6168
I20260619 08:56:28.299113 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 6302
I20260619 08:56:28.302541 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 5940
2026-06-19T08:56:28Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1755 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:56:28 dist-test-slave-h4g5 krb5kdc[6451](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:56:28 dist-test-slave-h4g5 krb5kdc[6451](info): set up 2 sockets
Jun 19 08:56:28 dist-test-slave-h4g5 krb5kdc[6451](info): commencing operation
krb5kdc: starting...
W20260619 08:56:29.256045 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 08:56:29 dist-test-slave-h4g5 krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:56:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:56:29Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260619 08:56:29.367908 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39107
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:29.446161 6464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:29.446370 6464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:29.446393 6464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:29.447517 6464 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:56:29.447566 6464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:29.447578 6464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:29.447589 6464 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:56:29.447599 6464 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:56:29.449081 6464 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:29.449326 6464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:29.449541 6464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:29.454109 6470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:29.454123 6472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:29.454114 6469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:29.454360 6464 server_base.cc:1034] running on GCE node
I20260619 08:56:29.454614 6464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:29.454903 6464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:29.456108 6464 hybrid_clock.cc:648] HybridClock initialized: now 1781859389456061 us; error 37 us; skew 500 ppm
Jun 19 08:56:29 dist-test-slave-h4g5 krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859389, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:29.458688 6464 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260619 08:56:29.459143 6464 webserver.cc:460] Webserver started at http://127.0.98.62:44987/ using document root <none> and password file <none>
I20260619 08:56:29.459342 6464 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:29.459385 6464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:29.459507 6464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:29.460623 6464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "ebd9ae8efff44fb09d1537d5c4147426"
format_stamp: "Formatted at 2026-06-19 08:56:29 on dist-test-slave-h4g5"
server_key: "2fde8ac0dde48c216073dac31c87dfbf"
server_key_iv: "ce6ddcd4aa1d61a1c9e271bda12b6d32"
server_key_version: "encryptionkey@0"
I20260619 08:56:29.460951 6464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "ebd9ae8efff44fb09d1537d5c4147426"
format_stamp: "Formatted at 2026-06-19 08:56:29 on dist-test-slave-h4g5"
server_key: "2fde8ac0dde48c216073dac31c87dfbf"
server_key_iv: "ce6ddcd4aa1d61a1c9e271bda12b6d32"
server_key_version: "encryptionkey@0"
I20260619 08:56:29.462173 6464 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:56:29.462971 6479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:29.463151 6464 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:29.463229 6464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "ebd9ae8efff44fb09d1537d5c4147426"
format_stamp: "Formatted at 2026-06-19 08:56:29 on dist-test-slave-h4g5"
server_key: "2fde8ac0dde48c216073dac31c87dfbf"
server_key_iv: "ce6ddcd4aa1d61a1c9e271bda12b6d32"
server_key_version: "encryptionkey@0"
I20260619 08:56:29.463297 6464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:29.480582 6464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:29.482861 6464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:29.482987 6464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:29.486831 6464 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39107
I20260619 08:56:29.486944 6531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39107 every 8 connection(s)
I20260619 08:56:29.487200 6464 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:56:29.488206 6532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:29.490453 6532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426: Bootstrap starting.
I20260619 08:56:29.491003 6532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:29.491262 6532 log.cc:824] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:29.491950 6532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426: No bootstrap required, opened a new log
I20260619 08:56:29.492278 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 6464
I20260619 08:56:29.492363 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 08:56:29.492537 392 external_mini_cluster.cc:1428] Setting key 05f4a0eaf7cea60b4a59f0e936adf595
I20260619 08:56:29.493718 6532 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } }
I20260619 08:56:29.493851 6532 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:29.493883 6532 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd9ae8efff44fb09d1537d5c4147426, State: Initialized, Role: FOLLOWER
I20260619 08:56:29.494041 6532 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } }
I20260619 08:56:29.494114 6532 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:56:29.494143 6532 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:56:29.494191 6532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:29.495018 6532 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } }
I20260619 08:56:29.495121 6532 leader_election.cc:302] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebd9ae8efff44fb09d1537d5c4147426; no voters:
I20260619 08:56:29.495266 6532 leader_election.cc:288] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:56:29.495301 6537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:29.495399 6537 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [term 1 LEADER]: Becoming Leader. State: Replica: ebd9ae8efff44fb09d1537d5c4147426, State: Running, Role: LEADER
I20260619 08:56:29.495476 6537 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } }
I20260619 08:56:29.495509 6532 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:56:29.495779 6537 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebd9ae8efff44fb09d1537d5c4147426. Latest consensus state: current_term: 1 leader_uuid: "ebd9ae8efff44fb09d1537d5c4147426" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } }
I20260619 08:56:29.495841 6537 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:29.495807 6539 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebd9ae8efff44fb09d1537d5c4147426" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } }
I20260619 08:56:29.495885 6539 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:29.496667 6542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:56:29.497233 6542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 08:56:29 dist-test-slave-h4g5 krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 08:56:29.500797 6542 catalog_manager.cc:1347] Generated new cluster ID: 8d8e9b83122a417ea361f872fee64a84
I20260619 08:56:29.500844 6542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:56:29.506345 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:29.493015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33266 (local address 127.0.98.62:39107)
0619 08:56:29.493193 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:29.493202 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:29.493226 (+ 24us) server_negotiation.cc:407] Connection header received
0619 08:56:29.493687 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:29.493693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:29.493739 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:29.493861 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:29.494534 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:29.495567 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:29.496679 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:29.497245 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:29.497343 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:29.499888 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:29.499892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:29.499892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:29.499901 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:29.504897 (+ 4996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:29.505223 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:29.505226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:29.505226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:29.505315 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:29.505484 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:29.505484 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:29.505485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:29.505720 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:29.505861 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:29.505967 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 08:56:29.506186 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":75,"threads_started":1}
I20260619 08:56:29.512585 6542 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:56:29.513098 6542 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:56:29.515290 6542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426: Generated new TSK 0
I20260619 08:56:29.515466 6542 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260619 08:56:29.543882 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39107
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:29.626061 6558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:29.626266 6558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:29.626289 6558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:29.627370 6558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:29.627403 6558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:29.627427 6558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 08:56:29.628888 6558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:29.629087 6558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:29.629267 6558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:29.633956 6566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:29.633945 6564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:29.633983 6563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:29.634179 6558 server_base.cc:1034] running on GCE node
I20260619 08:56:29.634409 6558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:29.634625 6558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:29.635761 6558 hybrid_clock.cc:648] HybridClock initialized: now 1781859389635747 us; error 33 us; skew 500 ppm
Jun 19 08:56:29 dist-test-slave-h4g5 krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859389, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:29.638166 6558 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 08:56:29.638542 6558 webserver.cc:460] Webserver started at http://127.0.98.1:40735/ using document root <none> and password file <none>
I20260619 08:56:29.638722 6558 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:29.638760 6558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:29.638849 6558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:29.639798 6558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-0/data/instance:
uuid: "b314603327ad4e21bf7d82da8a17eaa3"
format_stamp: "Formatted at 2026-06-19 08:56:29 on dist-test-slave-h4g5"
server_key: "c0a127a859a9ec885ff7026c9231356a"
server_key_iv: "e08b30ed29e0b014801cd4cc435aed7b"
server_key_version: "encryptionkey@0"
I20260619 08:56:29.640107 6558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b314603327ad4e21bf7d82da8a17eaa3"
format_stamp: "Formatted at 2026-06-19 08:56:29 on dist-test-slave-h4g5"
server_key: "c0a127a859a9ec885ff7026c9231356a"
server_key_iv: "e08b30ed29e0b014801cd4cc435aed7b"
server_key_version: "encryptionkey@0"
I20260619 08:56:29.641292 6558 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:56:29.641912 6573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:29.642187 6558 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:56:29.642267 6558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-0/wal
uuid: "b314603327ad4e21bf7d82da8a17eaa3"
format_stamp: "Formatted at 2026-06-19 08:56:29 on dist-test-slave-h4g5"
server_key: "c0a127a859a9ec885ff7026c9231356a"
server_key_iv: "e08b30ed29e0b014801cd4cc435aed7b"
server_key_version: "encryptionkey@0"
I20260619 08:56:29.642335 6558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:29.665686 6558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:29.667868 6558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:29.667986 6558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:29.668229 6558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:29.668536 6558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:29.668579 6558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:29.668603 6558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:29.668635 6558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:29.674089 6558 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37349
I20260619 08:56:29.674149 6686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37349 every 8 connection(s)
I20260619 08:56:29.674439 6558 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
I20260619 08:56:29.678809 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 6558
I20260619 08:56:29.678915 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-0/wal/instance
I20260619 08:56:29.679081 392 external_mini_cluster.cc:1428] Setting key ea8b0d827383c6a275dd2846b81b1f40
Jun 19 08:56:29 dist-test-slave-h4g5 krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859389, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 08:56:29.684671 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:29.675103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43111 (local address 127.0.98.62:39107)
0619 08:56:29.675166 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:29.675168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:29.675468 (+ 300us) server_negotiation.cc:407] Connection header received
0619 08:56:29.675977 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:29.675978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:29.675993 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:29.676049 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:29.677202 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:29.677744 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:29.678971 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:29.679231 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:29.679292 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:29.681536 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:29.681538 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:29.681538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:29.681547 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:29.683133 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:29.683522 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:29.683523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:29.683524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:29.683590 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:29.683921 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:29.683922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:29.683922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:29.684106 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:29.684174 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:29.684457 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 08:56:29.684553 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 08:56:29.684828 6689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:29.675090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.98.1:43111)
0619 08:56:29.675311 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:29.675352 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:29.675862 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:29.676221 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:29.676225 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:29.676856 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:29.677071 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:29.677077 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:29.677853 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:29.677853 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:29.678860 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:29.678863 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:29.679316 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:29.679317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:29.679459 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:29.680027 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:29.680038 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:29.681436 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:29.683225 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:29.683232 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:29.683234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:29.683463 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:29.683714 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:29.683715 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:29.683715 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:29.683851 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:29.684175 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:29.684181 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:29.684327 (+ 146us) client_negotiation.cc:769] Sending connection context
0619 08:56:29.684412 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 08:56:29.684555 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":102,"threads_started":1}
I20260619 08:56:29.685515 6687 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39107
I20260619 08:56:29.685616 6687 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:29.685817 6687 heartbeater.cc:511] Master 127.0.98.62:39107 requested a full tablet report, sending...
I20260619 08:56:29.686452 6496 ts_manager.cc:194] Registered new tserver with Master: b314603327ad4e21bf7d82da8a17eaa3 (127.0.98.1:37349)
I20260619 08:56:29.687129 6496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43111
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260619 08:56:29.709494 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39107
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:29.788615 6692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:29.788833 6692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:29.788856 6692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:29.789951 6692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:29.789983 6692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:29.790005 6692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 08:56:29.791361 6692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:29.791545 6692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:29.791723 6692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:29.796387 6697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:29.796434 6700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:29.796388 6698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:29.796518 6692 server_base.cc:1034] running on GCE node
I20260619 08:56:29.796842 6692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:29.797046 6692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:29.798199 6692 hybrid_clock.cc:648] HybridClock initialized: now 1781859389798186 us; error 30 us; skew 500 ppm
Jun 19 08:56:29 dist-test-slave-h4g5 krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859389, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:29.800642 6692 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 08:56:29.801048 6692 webserver.cc:460] Webserver started at http://127.0.98.2:38335/ using document root <none> and password file <none>
I20260619 08:56:29.801254 6692 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:29.801299 6692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:29.801396 6692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:29.802412 6692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-1/data/instance:
uuid: "14f33dcab292446f9ee74b2a2a56f56d"
format_stamp: "Formatted at 2026-06-19 08:56:29 on dist-test-slave-h4g5"
server_key: "3e4f609d70554d23a9898e5018f96d86"
server_key_iv: "2b64232258fbfb77bff21ef66e85bc37"
server_key_version: "encryptionkey@0"
I20260619 08:56:29.802718 6692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-1/wal/instance:
uuid: "14f33dcab292446f9ee74b2a2a56f56d"
format_stamp: "Formatted at 2026-06-19 08:56:29 on dist-test-slave-h4g5"
server_key: "3e4f609d70554d23a9898e5018f96d86"
server_key_iv: "2b64232258fbfb77bff21ef66e85bc37"
server_key_version: "encryptionkey@0"
I20260619 08:56:29.803900 6692 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:56:29.804685 6707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:29.804837 6692 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:29.804916 6692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-1/wal
uuid: "14f33dcab292446f9ee74b2a2a56f56d"
format_stamp: "Formatted at 2026-06-19 08:56:29 on dist-test-slave-h4g5"
server_key: "3e4f609d70554d23a9898e5018f96d86"
server_key_iv: "2b64232258fbfb77bff21ef66e85bc37"
server_key_version: "encryptionkey@0"
I20260619 08:56:29.804986 6692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:29.821734 6692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:29.823879 6692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:29.824000 6692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:29.824263 6692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:29.824558 6692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:29.824592 6692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:29.824621 6692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:29.824651 6692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:29.830574 6692 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42913
I20260619 08:56:29.830623 6820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42913 every 8 connection(s)
I20260619 08:56:29.830935 6692 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
I20260619 08:56:29.834475 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 6692
I20260619 08:56:29.834589 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-1/wal/instance
I20260619 08:56:29.834766 392 external_mini_cluster.cc:1428] Setting key 14654ab75a7f670983a3a47a32d347ac
Jun 19 08:56:29 dist-test-slave-h4g5 krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859389, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 08:56:29.840885 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:29.831611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50003 (local address 127.0.98.62:39107)
0619 08:56:29.831676 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:29.831678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:29.831879 (+ 201us) server_negotiation.cc:407] Connection header received
0619 08:56:29.832381 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:29.832383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:29.832392 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:29.832456 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:29.833394 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:29.833895 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:29.835157 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:29.835439 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:29.835506 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:29.838101 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:29.838103 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:29.838103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:29.838112 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:29.839488 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:29.839948 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:29.839949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:29.839949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:29.840004 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:29.840299 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:29.840300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:29.840300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:29.840399 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:29.840449 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:29.840761 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 08:56:29.840825 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 08:56:29.841164 6823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:29.831575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.98.2:50003)
0619 08:56:29.831758 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:29.831799 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:29.832297 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:29.832518 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:29.832521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:29.833097 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:29.833288 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:29.833293 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:29.833982 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:29.833982 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:29.835045 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:29.835048 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:29.835536 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:29.835537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:29.835678 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:29.836234 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:29.836244 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:29.838008 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:29.839568 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:29.839576 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:29.839578 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:29.839867 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:29.840069 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:29.840069 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:29.840070 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:29.840234 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:29.840459 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:29.840465 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:29.840626 (+ 161us) client_negotiation.cc:769] Sending connection context
0619 08:56:29.840744 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 08:56:29.840912 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":91,"threads_started":1}
I20260619 08:56:29.841710 6821 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39107
I20260619 08:56:29.841801 6821 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:29.841965 6821 heartbeater.cc:511] Master 127.0.98.62:39107 requested a full tablet report, sending...
I20260619 08:56:29.842299 6496 ts_manager.cc:194] Registered new tserver with Master: 14f33dcab292446f9ee74b2a2a56f56d (127.0.98.2:42913)
I20260619 08:56:29.842653 6496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50003
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 08:56:29.859254 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39107
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:29.936808 6826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:29.937000 6826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:29.937022 6826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:29.938091 6826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:29.938125 6826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:29.938149 6826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 08:56:29.939507 6826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:29.939685 6826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:29.939849 6826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:29.944540 6832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:29.944586 6826 server_base.cc:1034] running on GCE node
W20260619 08:56:29.944517 6831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:29.944563 6834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:29.944975 6826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:29.945216 6826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:29.946385 6826 hybrid_clock.cc:648] HybridClock initialized: now 1781859389946368 us; error 30 us; skew 500 ppm
Jun 19 08:56:29 dist-test-slave-h4g5 krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859389, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:29.953032 6826 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 08:56:29.953481 6826 webserver.cc:460] Webserver started at http://127.0.98.3:35529/ using document root <none> and password file <none>
I20260619 08:56:29.953701 6826 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:29.953744 6826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:29.953859 6826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:29.954905 6826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-2/data/instance:
uuid: "c2237f88ed954837b849c5727176aac3"
format_stamp: "Formatted at 2026-06-19 08:56:29 on dist-test-slave-h4g5"
server_key: "238278d48b8e2765f936978e9b693386"
server_key_iv: "3c93ceccf1855e5baaf00c43c18da35d"
server_key_version: "encryptionkey@0"
I20260619 08:56:29.955211 6826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c2237f88ed954837b849c5727176aac3"
format_stamp: "Formatted at 2026-06-19 08:56:29 on dist-test-slave-h4g5"
server_key: "238278d48b8e2765f936978e9b693386"
server_key_iv: "3c93ceccf1855e5baaf00c43c18da35d"
server_key_version: "encryptionkey@0"
I20260619 08:56:29.956475 6826 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:56:29.957315 6841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:29.957515 6826 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:56:29.957602 6826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-2/wal
uuid: "c2237f88ed954837b849c5727176aac3"
format_stamp: "Formatted at 2026-06-19 08:56:29 on dist-test-slave-h4g5"
server_key: "238278d48b8e2765f936978e9b693386"
server_key_iv: "3c93ceccf1855e5baaf00c43c18da35d"
server_key_version: "encryptionkey@0"
I20260619 08:56:29.957664 6826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:29.968782 6826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:29.970855 6826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:29.970999 6826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:29.971208 6826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:29.971483 6826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:29.971515 6826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:29.971542 6826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:29.971565 6826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:29.977036 6826 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39819
I20260619 08:56:29.977072 6954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39819 every 8 connection(s)
I20260619 08:56:29.977360 6826 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 08:56:29 dist-test-slave-h4g5 krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859389, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 08:56:29.983594 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 6826
I20260619 08:56:29.983681 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-2/wal/instance
I20260619 08:56:29.983860 392 external_mini_cluster.cc:1428] Setting key 09a852fea1a40d4fd31cbda4b14319ac
I20260619 08:56:29.985935 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:29.978002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46887 (local address 127.0.98.62:39107)
0619 08:56:29.978055 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:29.978057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:29.978236 (+ 179us) server_negotiation.cc:407] Connection header received
0619 08:56:29.978727 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:29.978729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:29.978739 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:29.978829 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:29.979718 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:29.980243 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:29.981163 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:29.981350 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:29.981406 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:29.983389 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:29.983390 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:29.983390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:29.983420 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:29.984822 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:29.985227 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:29.985228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:29.985228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:29.985284 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:29.985468 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:29.985469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:29.985469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:29.985566 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:29.985610 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:29.985815 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 08:56:29.985874 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 08:56:29.986104 6957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:29.977983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.98.3:46887)
0619 08:56:29.978127 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:29.978160 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:29.978638 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:29.978905 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:29.978908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:29.979446 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:29.979632 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:29.979636 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:29.980316 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:29.980317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:29.981090 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:29.981092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:29.981416 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:29.981417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:29.981533 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:29.981985 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:29.981993 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:29.983307 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:29.984888 (+ 1581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:29.984893 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:29.984895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:29.985093 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:29.985339 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:29.985339 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:29.985339 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:29.985425 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:29.985606 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:29.985608 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:29.985718 (+ 110us) client_negotiation.cc:769] Sending connection context
0619 08:56:29.985800 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 08:56:29.985918 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260619 08:56:29.986505 6955 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39107
I20260619 08:56:29.986589 6955 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:29.986752 6955 heartbeater.cc:511] Master 127.0.98.62:39107 requested a full tablet report, sending...
I20260619 08:56:29.987072 6496 ts_manager.cc:194] Registered new tserver with Master: c2237f88ed954837b849c5727176aac3 (127.0.98.3:39819)
I20260619 08:56:29.987411 6496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46887
I20260619 08:56:29.995563 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260619 08:56:30.023255 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46667
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39107,127.0.98.61:46667
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:30.112490 6960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:30.112722 6960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:30.112758 6960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:30.113934 6960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:56:30.113983 6960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:30.114006 6960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:30.114024 6960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:56:30.114049 6960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:56:30.115569 6960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39107,127.0.98.61:46667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
W20260619 08:56:30.115695 6960 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:39107,127.0.98.61:46667'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 08:56:30.115851 6960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:30.116070 6960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:30.120832 6965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:30.120908 6968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:30.120844 6966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:30.121076 6960 server_base.cc:1034] running on GCE node
I20260619 08:56:30.121297 6960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:30.121569 6960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:30.122706 6960 hybrid_clock.cc:648] HybridClock initialized: now 1781859390122666 us; error 57 us; skew 500 ppm
Jun 19 08:56:30 dist-test-slave-h4g5 krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859390, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:30.125098 6960 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260619 08:56:30.125577 6960 webserver.cc:460] Webserver started at http://127.0.98.61:35693/ using document root <none> and password file <none>
I20260619 08:56:30.125792 6960 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:30.125840 6960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:30.125963 6960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:30.126983 6960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-1/data/instance:
uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46"
format_stamp: "Formatted at 2026-06-19 08:56:30 on dist-test-slave-h4g5"
server_key: "f706c16ca168107ea1d1ef05a06635ff"
server_key_iv: "be73096ad7cd00cd5c2e3efff6fdf68f"
server_key_version: "encryptionkey@0"
I20260619 08:56:30.127314 6960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-1/wal/instance:
uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46"
format_stamp: "Formatted at 2026-06-19 08:56:30 on dist-test-slave-h4g5"
server_key: "f706c16ca168107ea1d1ef05a06635ff"
server_key_iv: "be73096ad7cd00cd5c2e3efff6fdf68f"
server_key_version: "encryptionkey@0"
I20260619 08:56:30.128568 6960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:56:30.129312 6975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:30.129498 6960 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:56:30.129577 6960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-1/wal
uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46"
format_stamp: "Formatted at 2026-06-19 08:56:30 on dist-test-slave-h4g5"
server_key: "f706c16ca168107ea1d1ef05a06635ff"
server_key_iv: "be73096ad7cd00cd5c2e3efff6fdf68f"
server_key_version: "encryptionkey@0"
I20260619 08:56:30.129637 6960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:30.142457 6960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:30.144608 6960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:30.144735 6960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:30.148370 6960 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46667
I20260619 08:56:30.148417 7027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46667 every 8 connection(s)
I20260619 08:56:30.148715 6960 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-1/data/info.pb
I20260619 08:56:30.149637 7028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:30.151100 7028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 08:56:30 dist-test-slave-h4g5 krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859390, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 08:56:30.158641 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 6960
I20260619 08:56:30.159780 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.151744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33272 (local address 127.0.98.62:39107)
0619 08:56:30.151860 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:30.151862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:30.151970 (+ 108us) server_negotiation.cc:407] Connection header received
0619 08:56:30.152444 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:30.152446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:30.152455 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:30.152515 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:30.153119 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.153609 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.154784 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.154981 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.155036 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.157035 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:30.157037 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:30.157037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:30.157044 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:30.158240 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.158641 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.158642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.158643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.158705 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.158992 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.158993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.158993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.159217 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:30.159288 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:30.159603 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 08:56:30.159690 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 08:56:30.159984 7030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.151700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.0.1:33272)
0619 08:56:30.151846 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:30.151887 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:30.152353 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:30.152572 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:30.152575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:30.152847 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:30.153031 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.153035 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.153703 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.153704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.154644 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.154647 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.155059 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.155060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.155168 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.155594 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:30.155602 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:30.156957 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:30.158315 (+ 1358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.158321 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.158322 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.158568 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.158788 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.158789 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.158789 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.158921 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.159293 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:30.159299 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:30.159460 (+ 161us) client_negotiation.cc:769] Sending connection context
0619 08:56:30.159563 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 08:56:30.159771 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":72,"threads_started":1}
I20260619 08:56:30.160610 7028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 08:56:30 dist-test-slave-h4g5 krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 19 08:56:30 dist-test-slave-h4g5 krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859390, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 08:56:30.167574 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.159207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55010 (local address 127.0.98.61:46667)
0619 08:56:30.159367 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:30.159372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:30.159388 (+ 16us) server_negotiation.cc:407] Connection header received
0619 08:56:30.159407 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:30.159409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:30.159421 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:30.159486 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:30.160116 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.160939 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.161787 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.162159 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.162208 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.164733 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:30.164740 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:30.164740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:30.164747 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:30.166361 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.166741 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.166743 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.166744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.166794 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.167013 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.167013 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.167014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.167140 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:30.167242 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:30.167371 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 08:56:30.167467 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
I20260619 08:56:30.168594 7030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.160830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46667 (local address 127.0.0.1:55014)
0619 08:56:30.160859 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:30.160868 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:30.160907 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:30.161103 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:30.161104 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:30.161269 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:30.161317 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.161318 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.161950 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.161950 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.162541 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.162543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.162926 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.162927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.162992 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.163288 (+ 296us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:30.163292 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:30.164922 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:30.167345 (+ 2423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.167346 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.167346 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.167578 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.167970 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.167971 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.167971 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.168031 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.168349 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:30.168350 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:30.168387 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 08:56:30.168455 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 08:56:30.168532 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14}
I20260619 08:56:30.168700 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.160873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55014 (local address 127.0.98.61:46667)
0619 08:56:30.161001 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:30.161002 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:30.161009 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:56:30.161025 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:30.161026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:30.161035 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:30.161086 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:30.161388 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.161868 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.162605 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.162857 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.162924 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.165030 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:30.165031 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:30.165032 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:30.165039 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:30.167267 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.167846 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.167847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.167848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.167908 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.168121 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.168122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.168122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.168267 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:30.168326 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:30.168471 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 08:56:30.168581 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":108,"thread_start_us":74,"threads_started":1}
I20260619 08:56:30.169864 7028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46: Bootstrap starting.
I20260619 08:56:30.170405 7028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:30.170655 7028 log.cc:824] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:30.171303 7028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46: No bootstrap required, opened a new log
I20260619 08:56:30.172783 7028 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } }
I20260619 08:56:30.172911 7028 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:30.172935 7028 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ff9d07cc16e4b48aff2249b4cdfdf46, State: Initialized, Role: FOLLOWER
I20260619 08:56:30.173025 7028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } }
I20260619 08:56:30.173280 7028 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:56:30.173236 7035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } } }
I20260619 08:56:30.173338 7035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 08:56:30.175048 7046 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 08:56:30.175101 7046 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 08:56:30.176474 6960 master_runner.cc:428] Detected that this master 3ff9d07cc16e4b48aff2249b4cdfdf46 is joining an existing cluster
I20260619 08:56:30.176530 6960 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:46667
I20260619 08:56:30.183480 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.177699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33280 (local address 127.0.98.62:39107)
0619 08:56:30.177747 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:30.177749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:30.177907 (+ 158us) server_negotiation.cc:407] Connection header received
0619 08:56:30.177934 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:30.177935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:30.177944 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:30.177993 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:30.178310 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.178835 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.179529 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.179720 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.179772 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.180707 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:30.180708 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:30.180709 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:30.180715 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:30.182180 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.182617 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.182618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.182619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.182698 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.182932 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.182933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.182933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.183071 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:30.183142 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:30.183325 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 08:56:30.183393 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 08:56:30.183544 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.177563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.0.1:33280)
0619 08:56:30.177856 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:30.177862 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:30.177896 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:30.178000 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:30.178000 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:30.178182 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:30.178246 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.178248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.178904 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.178905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.179459 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.179461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.179785 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.179786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.179842 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.180254 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:30.180258 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:30.180633 (+ 375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:30.182253 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.182254 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.182254 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.182526 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.182791 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.182792 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.182792 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.182856 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.183151 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:30.183152 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:30.183210 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 08:56:30.183295 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 08:56:30.183435 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":2,"thread_start_us":94,"threads_started":1}
I20260619 08:56:30.184820 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.177558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46667 (local address 127.0.0.1:55026)
0619 08:56:30.177737 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:30.177748 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:30.177806 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:30.177962 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:30.177963 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:30.178220 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:30.178285 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.178287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.179116 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.179116 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.179887 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.179890 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.180596 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.180597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.180666 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.181042 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:30.181047 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:30.181404 (+ 357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:30.183486 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.183486 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.183487 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.183892 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.184163 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.184164 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.184164 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.184238 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.184505 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:30.184505 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:30.184546 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 08:56:30.184623 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 08:56:30.184719 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":100,"threads_started":1}
I20260619 08:56:30.185101 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.177647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55026 (local address 127.0.98.61:46667)
0619 08:56:30.177691 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:30.177692 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:30.177813 (+ 121us) server_negotiation.cc:407] Connection header received
0619 08:56:30.177856 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:30.177857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:30.177868 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:30.177921 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:30.178375 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.179021 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.180231 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.180507 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.180571 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.181483 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:30.181485 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:30.181485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:30.181491 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:30.183416 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.183942 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.183943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.183943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.184035 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.184309 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.184310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.184310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.184430 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:30.184468 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:30.184647 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 08:56:30.185009 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":21}
Jun 19 08:56:30 dist-test-slave-h4g5 krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859389, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 08:56:30.192399 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.185988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55040 (local address 127.0.98.61:46667)
0619 08:56:30.186021 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:30.186023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:30.186176 (+ 153us) server_negotiation.cc:407] Connection header received
0619 08:56:30.186256 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:30.186257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:30.186266 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:30.186306 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:30.186693 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.187114 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.187834 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.188017 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.188069 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.190045 (+ 1976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:30.190046 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:30.190046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:30.190051 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:30.191217 (+ 1166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.191644 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.191644 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.191645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.191690 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.191904 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.191905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.191906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.192013 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:30.192074 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:30.192284 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 08:56:30.192340 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 08:56:30.192502 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.185965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46667 (local address 127.0.0.1:55040)
0619 08:56:30.186088 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:30.186121 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:30.186186 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:30.186319 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:30.186322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:30.186568 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:30.186622 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.186624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.187178 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.187179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.187762 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.187764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.188075 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.188076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.188191 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.188575 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:30.188580 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:30.189943 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:30.191291 (+ 1348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.191298 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.191300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.191568 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.191749 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.191749 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.191750 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.191819 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.192097 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:30.192104 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:30.192171 (+ 67us) client_negotiation.cc:769] Sending connection context
0619 08:56:30.192266 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 08:56:30.192393 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":76,"threads_started":1}
I20260619 08:56:30.193053 6496 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:46667: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 46667
}
attrs {
promote: true
}
}
dest_uuid: "ebd9ae8efff44fb09d1537d5c4147426"
cas_config_opid_index: -1
I20260619 08:56:30.193358 6496 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: true } }
I20260619 08:56:30.193840 7056 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3ff9d07cc16e4b48aff2249b4cdfdf46 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: true } } }
I20260619 08:56:30.194082 7002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:30.194674 7056 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:46667
I20260619 08:56:30.194758 7056 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ebd9ae8efff44fb09d1537d5c4147426" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: true } } }
I20260619 08:56:30.194819 7056 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:30.194922 6960 master.cc:537] Master@127.0.98.61:46667 shutting down...
I20260619 08:56:30.195014 7002 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [term 1 FOLLOWER]: Refusing update from remote peer ebd9ae8efff44fb09d1537d5c4147426: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 08:56:30.196722 7055 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.001s
I20260619 08:56:30.197734 6960 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:56:30.197821 6960 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:30.197870 6960 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46: stopping tablet replica
W20260619 08:56:30.198082 6481 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260619 08:56:30.198518 6481 consensus_peers.cc:487] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 -> Peer 3ff9d07cc16e4b48aff2249b4cdfdf46 (127.0.98.61:46667): Couldn't send request to peer 3ff9d07cc16e4b48aff2249b4cdfdf46. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 08:56:30.200275 6960 master.cc:552] Master@127.0.98.61:46667 shutdown complete.
I20260619 08:56:30.200310 6960 master_runner.cc:305] Clearing existing system tablet
I20260619 08:56:30.200436 6960 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:30.200475 6960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:30.201012 6960 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:30.201437 7060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:30.201536 6960 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:30.201583 6960 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:30.201603 6960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-1/wal
uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46"
format_stamp: "Formatted at 2026-06-19 08:56:30 on dist-test-slave-h4g5"
server_key: "f706c16ca168107ea1d1ef05a06635ff"
server_key_iv: "be73096ad7cd00cd5c2e3efff6fdf68f"
server_key_version: "encryptionkey@0"
I20260619 08:56:30.202126 6960 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:30.203373 6960 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 08:56:30.203425 6960 log.cc:1196] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 08:56:30.203595 6960 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46: Deleting consensus metadata
I20260619 08:56:30.203691 6960 master_runner.cc:315] Copying system tablet from 127.0.98.62:39107
I20260619 08:56:30.204277 6960 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:39107
I20260619 08:56:30.210577 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.204686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33290 (local address 127.0.98.62:39107)
0619 08:56:30.204741 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:30.204743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:30.204816 (+ 73us) server_negotiation.cc:407] Connection header received
0619 08:56:30.204839 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:30.204840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:30.204849 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:30.204890 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:30.205272 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.205821 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.206861 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.207076 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.207126 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.208258 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:30.208260 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:30.208260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:30.208269 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:30.209550 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.209926 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.209926 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.209927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.209973 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.210148 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.210148 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.210149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.210244 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:30.210290 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:30.210454 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 08:56:30.210511 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 08:56:30.210793 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.204576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.0.1:33290)
0619 08:56:30.204703 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:30.204714 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:30.204774 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:30.204914 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:30.204915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:30.205120 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:30.205199 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.205202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.205899 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.205899 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.206759 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.206762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.207150 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.207152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.207224 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.207653 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:30.207658 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:30.208167 (+ 509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:30.209624 (+ 1457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.209625 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.209626 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.209857 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.210026 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.210027 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.210027 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.210092 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.210295 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:30.210295 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:30.210351 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 08:56:30.210437 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 08:56:30.210550 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":67,"threads_started":1}
I20260619 08:56:30.211185 6516 tablet_copy_service.cc:140] P ebd9ae8efff44fb09d1537d5c4147426: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3ff9d07cc16e4b48aff2249b4cdfdf46 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33290)
I20260619 08:56:30.211297 6516 tablet_copy_service.cc:161] P ebd9ae8efff44fb09d1537d5c4147426: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3ff9d07cc16e4b48aff2249b4cdfdf46 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33290: session id = 3ff9d07cc16e4b48aff2249b4cdfdf46-00000000000000000000000000000000
I20260619 08:56:30.212195 6516 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 08:56:30.212834 6960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 08:56:30.213740 6960 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46: tablet copy: Starting download of 0 data blocks...
I20260619 08:56:30.213871 6960 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46: tablet copy: Starting download of 1 WAL segments...
I20260619 08:56:30.214712 6960 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 08:56:30.215487 6516 tablet_copy_service.cc:342] P ebd9ae8efff44fb09d1537d5c4147426: Request end of tablet copy session 3ff9d07cc16e4b48aff2249b4cdfdf46-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33290
I20260619 08:56:30.215560 6516 tablet_copy_service.cc:433] P ebd9ae8efff44fb09d1537d5c4147426: ending tablet copy session 3ff9d07cc16e4b48aff2249b4cdfdf46-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3ff9d07cc16e4b48aff2249b4cdfdf46
I20260619 08:56:30.218230 6960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:30.219511 7067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:30.219525 7068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:30.221071 7070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:30.221164 6960 server_base.cc:1034] running on GCE node
I20260619 08:56:30.221259 6960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:30.221406 6960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:30.222527 6960 hybrid_clock.cc:648] HybridClock initialized: now 1781859390222515 us; error 22 us; skew 500 ppm
Jun 19 08:56:30 dist-test-slave-h4g5 krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859390, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:30.223883 6960 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260619 08:56:30.224200 6960 webserver.cc:460] Webserver started at http://127.0.98.61:41129/ using document root <none> and password file <none>
I20260619 08:56:30.224323 6960 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:30.224362 6960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:30.225109 6960 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:56:30.225694 7077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:30.225852 6960 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 08:56:30.225900 6960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-1/wal
uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46"
format_stamp: "Formatted at 2026-06-19 08:56:30 on dist-test-slave-h4g5"
server_key: "f706c16ca168107ea1d1ef05a06635ff"
server_key_iv: "be73096ad7cd00cd5c2e3efff6fdf68f"
server_key_version: "encryptionkey@0"
I20260619 08:56:30.225955 6960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:30.239506 6960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:30.241078 6960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:30.243827 6960 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46667
I20260619 08:56:30.244136 7129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46667 every 8 connection(s)
I20260619 08:56:30.244234 6960 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-1/data/info.pb
I20260619 08:56:30.244937 7130 sys_catalog.cc:263] Verifying existing consensus state
I20260619 08:56:30.245306 7130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46: Bootstrap starting.
I20260619 08:56:30.247506 7130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 08:56:30.247690 7130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46: Bootstrap complete.
I20260619 08:56:30.247874 7130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: true } }
I20260619 08:56:30.247941 7130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3ff9d07cc16e4b48aff2249b4cdfdf46, State: Initialized, Role: LEARNER
I20260619 08:56:30.247967 7130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: true } }
I20260619 08:56:30.248132 7130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:56:30.248181 7133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: true } } }
I20260619 08:56:30.248291 7133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [sys.catalog]: This master's current role is: LEARNER
I20260619 08:56:30.249369 7144 catalog_manager.cc:1259] Loaded cluster ID: 8d8e9b83122a417ea361f872fee64a84
I20260619 08:56:30.249419 7144 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46: loading cluster ID for follower catalog manager: success
I20260619 08:56:30.250373 7144 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46: acquiring CA information for follower catalog manager: success
I20260619 08:56:30.250622 7144 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 08:56:30.275169 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.269113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55046 (local address 127.0.98.61:46667)
0619 08:56:30.269241 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:30.269242 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:30.269249 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:56:30.269273 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:30.269274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:30.269283 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:30.269337 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:30.269878 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.270538 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.271261 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.271460 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.271513 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.272758 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:30.272759 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:30.272760 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:30.272765 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:30.274141 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.274468 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.274469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.274469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.274520 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.274680 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.274681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.274681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.274786 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:30.274872 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:30.274971 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 08:56:30.275048 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260619 08:56:30.296561 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41911
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39107,127.0.98.61:46667,127.0.98.60:41911
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:30.374855 7148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:30.375052 7148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:30.375072 7148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:30.376183 7148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:56:30.376221 7148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:30.376235 7148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:30.376245 7148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:56:30.376255 7148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:56:30.377713 7148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39107,127.0.98.61:46667,127.0.98.60:41911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:30.377907 7148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:30.378075 7148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:30.382529 7154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:30.382643 7153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:30.382718 7148 server_base.cc:1034] running on GCE node
W20260619 08:56:30.382661 7156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:30.383011 7148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:30.383249 7148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:30.384405 7148 hybrid_clock.cc:648] HybridClock initialized: now 1781859390384393 us; error 28 us; skew 500 ppm
Jun 19 08:56:30 dist-test-slave-h4g5 krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859390, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:30.386797 7148 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260619 08:56:30.387183 7148 webserver.cc:460] Webserver started at http://127.0.98.60:33895/ using document root <none> and password file <none>
I20260619 08:56:30.387382 7148 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:30.387423 7148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:30.387535 7148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:30.388561 7148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-2/data/instance:
uuid: "60b4147da50c4ac78931f669afe3674e"
format_stamp: "Formatted at 2026-06-19 08:56:30 on dist-test-slave-h4g5"
server_key: "652a1770687a7a2eb67f2e83cc9d2c63"
server_key_iv: "202e793be06fe280c14208f25e813ad2"
server_key_version: "encryptionkey@0"
I20260619 08:56:30.388863 7148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-2/wal/instance:
uuid: "60b4147da50c4ac78931f669afe3674e"
format_stamp: "Formatted at 2026-06-19 08:56:30 on dist-test-slave-h4g5"
server_key: "652a1770687a7a2eb67f2e83cc9d2c63"
server_key_iv: "202e793be06fe280c14208f25e813ad2"
server_key_version: "encryptionkey@0"
I20260619 08:56:30.390048 7148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:56:30.390769 7163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:30.390955 7148 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:30.391036 7148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-2/wal
uuid: "60b4147da50c4ac78931f669afe3674e"
format_stamp: "Formatted at 2026-06-19 08:56:30 on dist-test-slave-h4g5"
server_key: "652a1770687a7a2eb67f2e83cc9d2c63"
server_key_iv: "202e793be06fe280c14208f25e813ad2"
server_key_version: "encryptionkey@0"
I20260619 08:56:30.391100 7148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:30.424676 7148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:30.426784 7148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:30.426962 7148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:30.431020 7148 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41911
I20260619 08:56:30.431066 7215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41911 every 8 connection(s)
I20260619 08:56:30.431381 7148 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-2/data/info.pb
I20260619 08:56:30.432308 7216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:30.434165 7216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 08:56:30 dist-test-slave-h4g5 krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859390, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 08:56:30.441044 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 7148
I20260619 08:56:30.442781 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.434988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33296 (local address 127.0.98.62:39107)
0619 08:56:30.435041 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:30.435043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:30.435155 (+ 112us) server_negotiation.cc:407] Connection header received
0619 08:56:30.435647 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:30.435648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:30.435660 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:30.435727 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:30.436423 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.436954 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.437907 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.438122 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.438172 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.440184 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:30.440186 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:30.440186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:30.440194 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:30.441473 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.441854 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.441855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.441856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.441916 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.442154 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.442155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.442155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.442251 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:30.442294 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:30.442579 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 08:56:30.442656 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 08:56:30.442890 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.434863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.0.1:33296)
0619 08:56:30.435036 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:30.435077 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:30.435527 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:30.435825 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:30.435829 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:30.436113 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:30.436327 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.436332 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.437028 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.437029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.437829 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.437831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.438177 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.438178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.438280 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.438693 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:30.438702 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:30.440066 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:30.441545 (+ 1479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.441550 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.441552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.441793 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.441966 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.441966 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.441967 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.442094 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.442304 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:30.442309 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:30.442451 (+ 142us) client_negotiation.cc:769] Sending connection context
0619 08:56:30.442537 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 08:56:30.442682 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":77,"threads_started":1}
I20260619 08:56:30.443605 7216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 08:56:30 dist-test-slave-h4g5 krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 19 08:56:30 dist-test-slave-h4g5 krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859390, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 08:56:30.449572 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.441529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40948 (local address 127.0.98.60:41911)
0619 08:56:30.441668 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:30.441672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:30.441685 (+ 13us) server_negotiation.cc:407] Connection header received
0619 08:56:30.441708 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:30.441710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:30.441720 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:30.441780 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:30.442397 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.443180 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.443971 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.444316 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.444374 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.446537 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:30.446544 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:30.446545 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:30.446551 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:30.448382 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.448702 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.448706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.448706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.448766 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.448937 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.448937 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.448938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.449118 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:30.449272 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:30.449355 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 08:56:30.449442 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":74,"threads_started":1}
I20260619 08:56:30.450403 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.443830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46667 (local address 127.0.0.1:55060)
0619 08:56:30.443868 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:30.443877 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:30.443910 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:30.444045 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:30.444046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:30.444195 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:30.444235 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.444237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.445036 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.445037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.445616 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.445618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.445954 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.445955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.446017 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.446477 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:30.446481 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:30.447873 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:30.449464 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.449465 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.449465 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.449699 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.449889 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.449890 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.449890 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.449933 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.450163 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:30.450164 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:30.450215 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 08:56:30.450277 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 08:56:30.450346 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 08:56:30.450419 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.443899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55060 (local address 127.0.98.61:46667)
0619 08:56:30.443948 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:30.443950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:30.443958 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:56:30.443982 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:30.443983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:30.443994 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:30.444039 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:30.444357 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.444948 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.445683 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.445899 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.445952 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.448003 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:30.448004 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:30.448004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:30.448010 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:30.449382 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.449773 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.449774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.449775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.449826 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.449987 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.449988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.449989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.450095 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:30.450159 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:30.450296 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 08:56:30.450358 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 08:56:30.450804 7216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41911 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 08:56:30 dist-test-slave-h4g5 krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859390, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 08:56:30.456276 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.451020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41911 (local address 127.0.0.1:40956)
0619 08:56:30.451057 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:30.451066 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:30.451098 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:30.451212 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:30.451213 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:30.451317 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:30.451344 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.451351 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.451925 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.451926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.452439 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.452440 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.452761 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.452762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.452807 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.453147 (+ 340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:30.453151 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:30.454209 (+ 1058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:30.455368 (+ 1159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.455368 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.455369 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.455505 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.455641 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.455641 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.455641 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.455679 (+ 38us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.455994 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:30.455995 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:30.456032 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 08:56:30.456131 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 08:56:30.456209 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260619 08:56:30.456336 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.451064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40956 (local address 127.0.98.60:41911)
0619 08:56:30.451097 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:30.451098 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:30.451107 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:56:30.451131 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:30.451132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:30.451142 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:30.451207 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:30.451401 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.451858 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.452495 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.452682 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.452737 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.454288 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:30.454289 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:30.454290 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:30.454295 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:30.455304 (+ 1009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.455551 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.455552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.455552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.455596 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.455714 (+ 118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.455714 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.455715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.455819 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:30.455976 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:30.456162 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 08:56:30.456256 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":16}
I20260619 08:56:30.457551 7216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e: Bootstrap starting.
I20260619 08:56:30.458333 7216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:30.458572 7216 log.cc:824] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:30.459185 7216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e: No bootstrap required, opened a new log
I20260619 08:56:30.460695 7216 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41911 } }
I20260619 08:56:30.460842 7216 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:30.460870 7216 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b4147da50c4ac78931f669afe3674e, State: Initialized, Role: FOLLOWER
I20260619 08:56:30.460971 7216 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41911 } }
I20260619 08:56:30.461196 7221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41911 } } }
I20260619 08:56:30.461244 7216 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:56:30.461272 7221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [sys.catalog]: This master's current role is: FOLLOWER
W20260619 08:56:30.462656 7232 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 08:56:30.462713 7232 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 08:56:30.465082 7148 master_runner.cc:428] Detected that this master 60b4147da50c4ac78931f669afe3674e is joining an existing cluster
I20260619 08:56:30.465137 7148 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:41911
I20260619 08:56:30.500836 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 6558
I20260619 08:56:30.504993 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37349
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39107
--webserver_port=40735
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:39107,127.0.98.61:46667
--tserver_master_addrs=127.0.98.62:39107,127.0.98.61:46667,127.0.98.60:41911 with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 08:56:30.512538 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.466259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40958 (local address 127.0.98.60:41911)
0619 08:56:30.466294 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:30.466296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:30.466740 (+ 444us) server_negotiation.cc:407] Connection header received
0619 08:56:30.466759 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:30.466760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:30.466778 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:30.466822 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:30.467353 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.468025 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.469033 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.469325 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.469361 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.509381 (+ 40020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:30.509382 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:30.509383 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:30.509389 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:30.511275 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.511840 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.511840 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.511841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.511884 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.512035 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.512035 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.512035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.512175 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:30.512213 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:30.512370 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 08:56:30.512444 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 08:56:30.512838 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.466330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41911 (local address 127.0.0.1:40958)
0619 08:56:30.466576 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:30.466583 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:30.466622 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:30.466855 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:30.466856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:30.467148 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:30.467247 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.467250 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.468145 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.468146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.468949 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.468951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.469440 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.469441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.469494 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.508647 (+ 39153us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:30.508653 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:30.509037 (+ 384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:30.511571 (+ 2534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.511571 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.511572 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.511760 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.511938 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.511939 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.511939 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.511981 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.512248 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:30.512249 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:30.512282 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 08:56:30.512651 (+ 369us) client_negotiation.cc:240] Negotiation successful
0619 08:56:30.512742 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":75,"threads_started":1}
I20260619 08:56:30.513018 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.466316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.0.1:33308)
0619 08:56:30.466476 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:30.466485 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:30.466564 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:30.466837 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:30.466838 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:30.467147 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:30.467706 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.467708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.468639 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.468640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.469405 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.469407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.469839 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.469840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.469894 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.470538 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:30.470541 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:30.508679 (+ 38138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:30.510831 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.510832 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.510833 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.511383 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.511613 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.511613 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.511614 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.511673 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.512686 (+ 1013us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:30.512687 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:30.512730 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 08:56:30.512797 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 08:56:30.512896 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":55,"threads_started":1}
I20260619 08:56:30.513362 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.466230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33308 (local address 127.0.98.62:39107)
0619 08:56:30.466282 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:30.466284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:30.466537 (+ 253us) server_negotiation.cc:407] Connection header received
0619 08:56:30.466638 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:30.466639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:30.466652 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:30.466704 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:30.467795 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.468555 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.469504 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.469733 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.469770 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.508830 (+ 39060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:30.508832 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:30.508832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:30.508842 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:30.510476 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.511483 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.511484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.511485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.511543 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.512454 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.512455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.512455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.512596 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:30.512657 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:30.513197 (+ 540us) server_negotiation.cc:299] Negotiation successful
0619 08:56:30.513276 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 08:56:30.514591 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.466398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55074 (local address 127.0.98.61:46667)
0619 08:56:30.466436 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:30.466437 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:30.466537 (+ 100us) server_negotiation.cc:407] Connection header received
0619 08:56:30.466722 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:30.466724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:30.466733 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:30.466778 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:30.467299 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.467866 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.468914 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.469215 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.469251 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.510577 (+ 41326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:30.510579 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:30.510580 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:30.510588 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:30.512364 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.513643 (+ 1279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.513645 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.513645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.513699 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.513886 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.513887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.513888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.514011 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:30.514049 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:30.514419 (+ 370us) server_negotiation.cc:299] Negotiation successful
0619 08:56:30.514501 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 08:56:30.515232 7238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.466194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46667 (local address 127.0.0.1:55074)
0619 08:56:30.466456 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:30.466467 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:30.466661 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:30.466839 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:30.466840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:30.467158 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:30.467227 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.467229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.467942 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.467943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.468831 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.468833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.469334 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.469334 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.469390 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.469898 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:30.469902 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:30.509742 (+ 39840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:30.513367 (+ 3625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.513368 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.513368 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.513559 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.513763 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.513764 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.513764 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.513824 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.514089 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:30.514089 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:30.514306 (+ 217us) client_negotiation.cc:769] Sending connection context
0619 08:56:30.514891 (+ 585us) client_negotiation.cc:240] Negotiation successful
0619 08:56:30.514988 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":71,"threads_started":1}
Jun 19 08:56:30 dist-test-slave-h4g5 krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859389, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 08:56:30.522514 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.515548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40970 (local address 127.0.98.60:41911)
0619 08:56:30.515688 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:30.515690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:30.515696 (+ 6us) server_negotiation.cc:407] Connection header received
0619 08:56:30.515718 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:30.515719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:30.515727 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:30.515780 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:30.516064 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.516791 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.517712 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.518021 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.518085 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.520033 (+ 1948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:30.520035 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:30.520035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:30.520041 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:30.521464 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.521833 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.521834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.521834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.521890 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.522055 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.522056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.522056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.522173 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:30.522233 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:30.522368 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 08:56:30.522435 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260619 08:56:30.522533 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.515461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41911 (local address 127.0.0.1:40970)
0619 08:56:30.515507 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:30.515518 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:30.515561 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:30.515803 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:30.515804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:30.515953 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:30.516000 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.516002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.516876 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.516877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.517646 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.517649 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.518091 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.518092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.518164 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.518538 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:30.518543 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:30.519954 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:30.521545 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.521546 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.521546 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.521762 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.521950 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.521951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.521951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.522005 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.522230 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:30.522231 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:30.522274 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 08:56:30.522336 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 08:56:30.522434 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 08:56:30.523286 6496 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:41911: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "60b4147da50c4ac78931f669afe3674e"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 41911
}
attrs {
promote: true
}
}
dest_uuid: "ebd9ae8efff44fb09d1537d5c4147426"
cas_config_opid_index: 5
I20260619 08:56:30.523518 6496 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: true } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: true } }
I20260619 08:56:30.523849 7055 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 60b4147da50c4ac78931f669afe3674e (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: true } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: true } } }
I20260619 08:56:30.524461 7190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:30.524848 7055 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:41911
I20260619 08:56:30.524915 7055 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ebd9ae8efff44fb09d1537d5c4147426" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: true } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: true } } }
I20260619 08:56:30.524973 7055 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:30.525279 7190 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [term 1 FOLLOWER]: Refusing update from remote peer ebd9ae8efff44fb09d1537d5c4147426: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 08:56:30.525460 7148 master.cc:537] Master@127.0.98.60:41911 shutting down...
I20260619 08:56:30.525527 7056 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 08:56:30.526222 6480 consensus_peers.cc:487] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 -> Peer 60b4147da50c4ac78931f669afe3674e (127.0.98.60:41911): Couldn't send request to peer 60b4147da50c4ac78931f669afe3674e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 08:56:30.528158 7148 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:56:30.528247 7148 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:30.528291 7148 tablet_replica.cc:330] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e: stopping tablet replica
I20260619 08:56:30.530336 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.525141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46667 (local address 127.0.0.1:55088)
0619 08:56:30.526549 (+ 1408us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:30.526560 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:30.526687 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:30.526920 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:30.526921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:30.526923 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:30.526961 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.526972 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.527860 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.527862 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.529197 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.529200 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.530087 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.530088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.530153 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.530178 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 08:56:30.530225 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 08:56:30.530265 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1393}
I20260619 08:56:30.530625 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.525234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55088 (local address 127.0.98.61:46667)
0619 08:56:30.526518 (+ 1284us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:30.526520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:30.526660 (+ 140us) server_negotiation.cc:407] Connection header received
0619 08:56:30.526766 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:30.526767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:30.526789 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:30.526839 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:30.527033 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.527762 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.529296 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.529918 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.529987 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.530054 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:30.530524 (+ 470us) server_negotiation.cc:299] Negotiation successful
0619 08:56:30.530564 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1260}
I20260619 08:56:30.531018 7104 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [term 1 LEARNER]: Refusing update from remote peer ebd9ae8efff44fb09d1537d5c4147426: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 08:56:30.531204 7056 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 08:56:30.531749 7104 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 60b4147da50c4ac78931f669afe3674e (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: true } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: true } } }
I20260619 08:56:30.532876 7133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebd9ae8efff44fb09d1537d5c4147426. Latest consensus state: current_term: 1 leader_uuid: "ebd9ae8efff44fb09d1537d5c4147426" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: true } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: true } } }
I20260619 08:56:30.532956 7133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [sys.catalog]: This master's current role is: LEARNER
I20260619 08:56:30.532989 7133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ebd9ae8efff44fb09d1537d5c4147426" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: true } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: true } } }
I20260619 08:56:30.533025 7133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [sys.catalog]: This master's current role is: LEARNER
I20260619 08:56:30.533025 7056 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426: attempting to promote NON_VOTER 3ff9d07cc16e4b48aff2249b4cdfdf46 to VOTER
I20260619 08:56:30.533200 7056 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: false } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: true } }
I20260619 08:56:30.533800 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.533672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41911 (local address 127.0.0.1:40980)
0619 08:56:30.533719 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:30.533764 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 08:56:30.533800 7104 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [term 1 LEARNER]: Refusing update from remote peer ebd9ae8efff44fb09d1537d5c4147426: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 08:56:30.534152 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.534072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41911 (local address 127.0.0.1:40982)
0619 08:56:30.534108 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:30.534130 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 08:56:30.534219 6480 consensus_peers.cc:487] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 -> Peer 60b4147da50c4ac78931f669afe3674e (127.0.98.60:41911): Couldn't send request to peer 60b4147da50c4ac78931f669afe3674e. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41911: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 08:56:30.534278 7244 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 08:56:30.534852 7055 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3ff9d07cc16e4b48aff2249b4cdfdf46 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: false } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: true } } }
I20260619 08:56:30.535037 7104 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3ff9d07cc16e4b48aff2249b4cdfdf46 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: false } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: true } } }
I20260619 08:56:30.535748 7055 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ebd9ae8efff44fb09d1537d5c4147426" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: false } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: true } } }
I20260619 08:56:30.535825 7055 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:30.535868 7055 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ebd9ae8efff44fb09d1537d5c4147426" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: false } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: true } } }
I20260619 08:56:30.535902 7055 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:30.535943 7243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ebd9ae8efff44fb09d1537d5c4147426" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: false } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: true } } }
I20260619 08:56:30.536023 7243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 08:56:30.541101 7148 master.cc:552] Master@127.0.98.60:41911 shutdown complete.
I20260619 08:56:30.541158 7148 master_runner.cc:305] Clearing existing system tablet
I20260619 08:56:30.541285 7148 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:30.541337 7148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:30.542078 7148 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:30.542589 7247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:30.542693 7148 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:30.542742 7148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:30.542769 7148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-2/wal
uuid: "60b4147da50c4ac78931f669afe3674e"
format_stamp: "Formatted at 2026-06-19 08:56:30 on dist-test-slave-h4g5"
server_key: "652a1770687a7a2eb67f2e83cc9d2c63"
server_key_iv: "202e793be06fe280c14208f25e813ad2"
server_key_version: "encryptionkey@0"
I20260619 08:56:30.543480 7148 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:30.544919 7148 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 08:56:30.544984 7148 log.cc:1196] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 08:56:30.545157 7148 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e: Deleting consensus metadata
I20260619 08:56:30.545269 7148 master_runner.cc:315] Copying system tablet from 127.0.98.62:39107
I20260619 08:56:30.545845 7148 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:39107
I20260619 08:56:30.553100 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.546198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33324 (local address 127.0.98.62:39107)
0619 08:56:30.546242 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:30.546244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:30.546345 (+ 101us) server_negotiation.cc:407] Connection header received
0619 08:56:30.546411 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:30.546412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:30.546422 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:30.546492 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:30.546847 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.547603 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.548754 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.549056 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.549118 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.550249 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:30.550251 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:30.550251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:30.550258 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:30.551892 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.552316 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.552318 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.552318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.552383 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.552586 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.552587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.552587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.552714 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:30.552783 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:30.552934 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 08:56:30.553011 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 08:56:30.553112 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.546133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.0.1:33324)
0619 08:56:30.546265 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:30.546276 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:30.546346 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:30.546511 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:30.546512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:30.546706 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:30.546760 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.546762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.547697 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.547698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.548668 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.548670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.549132 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.549133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.549207 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.549739 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:30.549744 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:30.550147 (+ 403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:30.551985 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.551986 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.551986 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.552239 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.552455 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.552456 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.552456 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.552525 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.552787 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:30.552788 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:30.552835 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 08:56:30.552906 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 08:56:30.553007 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260619 08:56:30.553723 6516 tablet_copy_service.cc:140] P ebd9ae8efff44fb09d1537d5c4147426: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 60b4147da50c4ac78931f669afe3674e ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:33324)
I20260619 08:56:30.553794 6516 tablet_copy_service.cc:161] P ebd9ae8efff44fb09d1537d5c4147426: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 60b4147da50c4ac78931f669afe3674e at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:33324: session id = 60b4147da50c4ac78931f669afe3674e-00000000000000000000000000000000
I20260619 08:56:30.554369 6516 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 08:56:30.555672 7148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 08:56:30.556936 7148 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e: tablet copy: Starting download of 0 data blocks...
I20260619 08:56:30.557077 7148 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e: tablet copy: Starting download of 1 WAL segments...
I20260619 08:56:30.557837 7148 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 08:56:30.558732 6516 tablet_copy_service.cc:342] P ebd9ae8efff44fb09d1537d5c4147426: Request end of tablet copy session 60b4147da50c4ac78931f669afe3674e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:33324
I20260619 08:56:30.558792 6516 tablet_copy_service.cc:433] P ebd9ae8efff44fb09d1537d5c4147426: ending tablet copy session 60b4147da50c4ac78931f669afe3674e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 60b4147da50c4ac78931f669afe3674e
I20260619 08:56:30.561467 7148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:30.562657 7254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:30.564946 7255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:30.565379 7257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:30.572240 7148 server_base.cc:1034] running on GCE node
I20260619 08:56:30.572355 7148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:30.572506 7148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:30.573642 7148 hybrid_clock.cc:648] HybridClock initialized: now 1781859390573623 us; error 38 us; skew 500 ppm
Jun 19 08:56:30 dist-test-slave-h4g5 krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859390, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:30.575137 7148 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260619 08:56:30.575397 7148 webserver.cc:460] Webserver started at http://127.0.98.60:42123/ using document root <none> and password file <none>
I20260619 08:56:30.575506 7148 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:30.575537 7148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:30.576311 7148 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:30.576911 7264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:30.577021 7148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:30.577059 7148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-2/wal
uuid: "60b4147da50c4ac78931f669afe3674e"
format_stamp: "Formatted at 2026-06-19 08:56:30 on dist-test-slave-h4g5"
server_key: "652a1770687a7a2eb67f2e83cc9d2c63"
server_key_iv: "202e793be06fe280c14208f25e813ad2"
server_key_version: "encryptionkey@0"
I20260619 08:56:30.577119 7148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:30.593333 7148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:30.595202 7148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:30.598634 7148 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41911
I20260619 08:56:30.598660 7316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41911 every 8 connection(s)
I20260619 08:56:30.599010 7148 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/master-2/data/info.pb
W20260619 08:56:30.603753 7241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:30.603960 7241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:30.603992 7241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 08:56:30.604885 7317 sys_catalog.cc:263] Verifying existing consensus state
I20260619 08:56:30.605386 7317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e: Bootstrap starting.
W20260619 08:56:30.605623 7241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:30.605669 7241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:30.605708 7241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 08:56:30.607796 7241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37349
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40735
--tserver_master_addrs=127.0.98.62:39107,127.0.98.61:46667,127.0.98.60:41911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:30.608052 7241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:30.608260 7317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 08:56:30.608347 7241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 08:56:30.608448 7317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e: Bootstrap complete.
I20260619 08:56:30.608616 7317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: false } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: true } }
I20260619 08:56:30.608703 7317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 60b4147da50c4ac78931f669afe3674e, State: Initialized, Role: LEARNER
I20260619 08:56:30.608738 7317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: false } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: true } }
I20260619 08:56:30.608999 7317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:56:30.608934 7323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: false } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: true } } }
I20260619 08:56:30.609063 7323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [sys.catalog]: This master's current role is: LEARNER
I20260619 08:56:30.610430 7337 catalog_manager.cc:1259] Loaded cluster ID: 8d8e9b83122a417ea361f872fee64a84
I20260619 08:56:30.610522 7337 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e: loading cluster ID for follower catalog manager: success
I20260619 08:56:30.611497 7337 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e: acquiring CA information for follower catalog manager: success
I20260619 08:56:30.611752 7337 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 08:56:30.614113 7333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:30.614126 7331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:30.614156 7339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:30.614454 7241 server_base.cc:1034] running on GCE node
I20260619 08:56:30.614609 7241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:30.614807 7241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:30.615973 7241 hybrid_clock.cc:648] HybridClock initialized: now 1781859390615954 us; error 26 us; skew 500 ppm
Jun 19 08:56:30 dist-test-slave-h4g5 krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859390, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:30.618468 7241 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 08:56:30.618896 7241 webserver.cc:460] Webserver started at http://127.0.98.1:40735/ using document root <none> and password file <none>
I20260619 08:56:30.619096 7241 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:30.619159 7241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:30.620463 7241 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:30.620956 7346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:30.621102 7241 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:30.621179 7241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-0/wal
uuid: "b314603327ad4e21bf7d82da8a17eaa3"
format_stamp: "Formatted at 2026-06-19 08:56:29 on dist-test-slave-h4g5"
server_key: "c0a127a859a9ec885ff7026c9231356a"
server_key_iv: "e08b30ed29e0b014801cd4cc435aed7b"
server_key_version: "encryptionkey@0"
I20260619 08:56:30.621518 7241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:30.636652 7241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:30.638422 7241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:30.638556 7241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:30.638764 7241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:30.639046 7241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:30.639078 7241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:30.639107 7241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:30.639130 7241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:30.644856 7241 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37349
I20260619 08:56:30.644929 7459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37349 every 8 connection(s)
I20260619 08:56:30.645229 7241 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
I20260619 08:56:30.649536 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 7241
I20260619 08:56:30.649650 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 6692
Jun 19 08:56:30 dist-test-slave-h4g5 krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859390, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 08:56:30 dist-test-slave-h4g5 krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859390, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 08:56:30.656657 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.646207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53659 (local address 127.0.98.62:39107)
0619 08:56:30.646253 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:30.646255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:30.646266 (+ 11us) server_negotiation.cc:407] Connection header received
0619 08:56:30.647279 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:30.647281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:30.647294 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:30.647353 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:30.648263 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.648803 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.650265 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.650557 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.650629 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.653738 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:30.653739 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:30.653739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:30.653748 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:30.655144 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.655519 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.655520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.655520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.655582 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.655878 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.655879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.655879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.656018 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:30.656062 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:30.656483 (+ 421us) server_negotiation.cc:299] Negotiation successful
0619 08:56:30.656566 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 08:56:30.656843 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.645916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41911 (local address 127.0.98.1:55779)
0619 08:56:30.646339 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:30.646349 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:30.647199 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:30.647484 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:30.647485 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:30.647922 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:30.648176 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.648182 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.650651 (+ 2469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.650652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.651356 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.651359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.651799 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.651800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.651876 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.652302 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:30.652306 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:30.653649 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:30.655556 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.655557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.655558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.655741 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.655932 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.655933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.655933 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.655981 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.656246 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:30.656247 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:30.656325 (+ 78us) client_negotiation.cc:769] Sending connection context
0619 08:56:30.656423 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 08:56:30.656596 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":6,"thread_start_us":96,"threads_started":1}
I20260619 08:56:30.657136 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.646227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55779 (local address 127.0.98.60:41911)
0619 08:56:30.646453 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:30.646455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:30.646462 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:56:30.647326 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:30.647327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:30.647341 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:30.647403 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:30.649799 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.650578 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.651441 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.651723 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.651793 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.654013 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:30.654014 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:30.654014 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:30.654021 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:30.655469 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.655822 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.655823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.655824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.655877 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.656031 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.656032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.656032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.656173 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:30.656224 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:30.656969 (+ 745us) server_negotiation.cc:299] Negotiation successful
0619 08:56:30.657049 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":54,"threads_started":1}
I20260619 08:56:30.657521 7465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.645916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.98.1:53659)
0619 08:56:30.646100 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:30.646142 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:30.647167 (+ 1025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:30.647480 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:30.647484 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:30.647919 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:30.648175 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.648181 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.648948 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.648949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.650159 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.650161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.650641 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.650642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.650792 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.651356 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:30.651366 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:30.653649 (+ 2283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:30.655217 (+ 1568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.655225 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.655227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.655451 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.655653 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.655653 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.655654 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.655786 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.656144 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:30.656152 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:30.656325 (+ 173us) client_negotiation.cc:769] Sending connection context
0619 08:56:30.657312 (+ 987us) client_negotiation.cc:240] Negotiation successful
0619 08:56:30.657421 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":1,"thread_start_us":88,"threads_started":1}
I20260619 08:56:30.658550 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42913
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39107
--webserver_port=38335
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:39107,127.0.98.61:46667
--tserver_master_addrs=127.0.98.62:39107,127.0.98.61:46667,127.0.98.60:41911 with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 08:56:30.659117 7460 heartbeater.cc:347] Connected to a master server at 127.0.98.60:41911
I20260619 08:56:30.659219 7460 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:30.659418 7460 heartbeater.cc:511] Master 127.0.98.60:41911 requested a full tablet report, sending...
I20260619 08:56:30.659835 7461 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39107
I20260619 08:56:30.659888 7461 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:30.659943 7461 heartbeater.cc:511] Master 127.0.98.62:39107 requested a full tablet report, sending...
I20260619 08:56:30.659940 7273 ts_manager.cc:194] Registered new tserver with Master: b314603327ad4e21bf7d82da8a17eaa3 (127.0.98.1:37349)
I20260619 08:56:30.660219 6496 ts_manager.cc:194] Re-registered known tserver with Master: b314603327ad4e21bf7d82da8a17eaa3 (127.0.98.1:37349)
I20260619 08:56:30.660705 6496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53659
Jun 19 08:56:30 dist-test-slave-h4g5 krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859390, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 08:56:30.664311 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.646238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49659 (local address 127.0.98.61:46667)
0619 08:56:30.646292 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:30.646294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:30.646575 (+ 281us) server_negotiation.cc:407] Connection header received
0619 08:56:30.647350 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:30.647351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:30.647363 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:30.647414 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:30.648943 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.649663 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.659294 (+ 9631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.659571 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.659612 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.661975 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:30.661977 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:30.661977 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:30.661984 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:30.663269 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.663618 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.663618 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.663619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.663662 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.663856 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.663856 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.663856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.663956 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:30.663999 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:30.664186 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 08:56:30.664251 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 08:56:30.664369 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.645916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46667 (local address 127.0.98.1:49659)
0619 08:56:30.646514 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:30.646521 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:30.647197 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:30.647481 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:30.647485 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:30.647924 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:30.648176 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.648182 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.658335 (+ 10153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.658336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.659184 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.659186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.659691 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.659692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.660125 (+ 433us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.660537 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:30.660541 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:30.661867 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:30.663345 (+ 1478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.663346 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.663346 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.663550 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.663711 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.663711 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.663712 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.663772 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.664006 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:30.664007 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:30.664060 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 08:56:30.664146 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 08:56:30.664255 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":41,"thread_start_us":69,"threads_started":1}
I20260619 08:56:30.664947 7462 heartbeater.cc:347] Connected to a master server at 127.0.98.61:46667
I20260619 08:56:30.664994 7462 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:30.665052 7462 heartbeater.cc:511] Master 127.0.98.61:46667 requested a full tablet report, sending...
I20260619 08:56:30.665378 7094 ts_manager.cc:194] Registered new tserver with Master: b314603327ad4e21bf7d82da8a17eaa3 (127.0.98.1:37349)
W20260619 08:56:30.739777 7468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:30.739980 7468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:30.740010 7468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:30.741160 7468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:30.741202 7468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:30.741237 7468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 08:56:30.742777 7468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42913
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38335
--tserver_master_addrs=127.0.98.62:39107,127.0.98.61:46667,127.0.98.60:41911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:30.743006 7468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:30.743206 7468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:30.748389 7476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:30.748404 7474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:30.748413 7473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:30.748546 7468 server_base.cc:1034] running on GCE node
I20260619 08:56:30.748864 7468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:30.749116 7468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:30.750272 7468 hybrid_clock.cc:648] HybridClock initialized: now 1781859390750265 us; error 36 us; skew 500 ppm
Jun 19 08:56:30 dist-test-slave-h4g5 krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859390, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:30.752770 7468 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 08:56:30.753212 7468 webserver.cc:460] Webserver started at http://127.0.98.2:38335/ using document root <none> and password file <none>
I20260619 08:56:30.753433 7468 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:30.753496 7468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:30.754796 7468 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:30.755332 7483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:30.755486 7468 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:30.755565 7468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-1/wal
uuid: "14f33dcab292446f9ee74b2a2a56f56d"
format_stamp: "Formatted at 2026-06-19 08:56:29 on dist-test-slave-h4g5"
server_key: "3e4f609d70554d23a9898e5018f96d86"
server_key_iv: "2b64232258fbfb77bff21ef66e85bc37"
server_key_version: "encryptionkey@0"
I20260619 08:56:30.755895 7468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:30.776130 7468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:30.778244 7468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:30.778391 7468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:30.778621 7468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:30.778927 7468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:30.778963 7468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:30.778996 7468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:30.779021 7468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:30.784976 7468 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42913
I20260619 08:56:30.785037 7596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42913 every 8 connection(s)
I20260619 08:56:30.785372 7468 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 08:56:30 dist-test-slave-h4g5 krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859390, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 19 08:56:30 dist-test-slave-h4g5 krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859390, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 19 08:56:30 dist-test-slave-h4g5 krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859390, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 08:56:30.794268 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 7468
I20260619 08:56:30.794409 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 6826
I20260619 08:56:30.797293 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.786440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50347 (local address 127.0.98.61:46667)
0619 08:56:30.786482 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:30.786484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:30.786527 (+ 43us) server_negotiation.cc:407] Connection header received
0619 08:56:30.787467 (+ 940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:30.787468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:30.787480 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:30.787533 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:30.788269 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.789025 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.790733 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.790947 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.790999 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.793119 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:30.793120 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:30.793121 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:30.793127 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:30.795957 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.796325 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.796326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.796327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.796387 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.796639 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.796641 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.796641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.796762 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:30.796821 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:30.797126 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 08:56:30.797204 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 08:56:30.797297 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.786068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47121 (local address 127.0.98.62:39107)
0619 08:56:30.786122 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:30.786124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:30.786507 (+ 383us) server_negotiation.cc:407] Connection header received
0619 08:56:30.787520 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:30.787522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:30.787534 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:30.787582 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:30.788220 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.788999 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.790437 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.790705 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.790761 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.793979 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:30.793980 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:30.793980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:30.793987 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:30.795488 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.795927 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.795928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.795928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.795985 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.796515 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.796515 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.796516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.796719 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:30.796777 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:30.797127 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 08:56:30.797206 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 08:56:30.797482 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.786072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59209 (local address 127.0.98.60:41911)
0619 08:56:30.786156 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:30.786158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:30.786520 (+ 362us) server_negotiation.cc:407] Connection header received
0619 08:56:30.787633 (+ 1113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:30.787634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:30.787646 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:30.787694 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:30.789109 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.789584 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.790343 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.790606 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.790667 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.793576 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:30.793578 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:30.793578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:30.793584 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:30.796178 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.796514 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.796516 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.796516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.796577 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.796942 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.796943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.796943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.797061 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:30.797100 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:30.797313 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 08:56:30.797389 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 08:56:30.797789 7601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.786060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.98.2:47121)
0619 08:56:30.786250 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:30.786284 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:30.787342 (+ 1058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:30.787608 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:30.787613 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:30.787911 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:30.788144 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.788148 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.789132 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.789134 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.790348 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.790351 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.790799 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.790800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.790859 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.791447 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:30.791450 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:30.793906 (+ 2456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:30.795575 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.795581 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.795584 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.795843 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.796051 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.796052 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.796052 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.796437 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.796844 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:30.796845 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:30.796979 (+ 134us) client_negotiation.cc:769] Sending connection context
0619 08:56:30.797417 (+ 438us) client_negotiation.cc:240] Negotiation successful
0619 08:56:30.797571 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":41,"spinlock_wait_cycles":6912,"thread_start_us":88,"threads_started":1}
I20260619 08:56:30.797806 7602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.786061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41911 (local address 127.0.98.2:59209)
0619 08:56:30.786454 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:30.786464 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:30.787348 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:30.787713 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:30.787714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:30.787953 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:30.788144 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.788150 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.789656 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.789657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.790269 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.790271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.790677 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.790678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.790806 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.791397 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:30.791407 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:30.793458 (+ 2051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:30.796258 (+ 2800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.796258 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.796259 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.796437 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.796820 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.796821 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.796821 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.796876 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.797142 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:30.797143 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:30.797211 (+ 68us) client_negotiation.cc:769] Sending connection context
0619 08:56:30.797551 (+ 340us) client_negotiation.cc:240] Negotiation successful
0619 08:56:30.797624 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":4,"thread_start_us":90,"threads_started":1}
I20260619 08:56:30.798182 7598 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39107
I20260619 08:56:30.798267 7598 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:30.798426 7598 heartbeater.cc:511] Master 127.0.98.62:39107 requested a full tablet report, sending...
I20260619 08:56:30.798769 6496 ts_manager.cc:194] Re-registered known tserver with Master: 14f33dcab292446f9ee74b2a2a56f56d (127.0.98.2:42913)
I20260619 08:56:30.799129 6496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47121
I20260619 08:56:30.799418 7603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.786356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46667 (local address 127.0.98.2:50347)
0619 08:56:30.786474 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:30.786483 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:30.787350 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:30.787958 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:30.787959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:30.788116 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:30.788163 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.788165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.789138 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.789139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.790351 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.790352 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.791011 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.791012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.791080 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.791549 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:30.791553 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:30.793038 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:30.796030 (+ 2992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.796031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.796031 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.796266 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.796459 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.796460 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.796460 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.796525 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.796818 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:30.796824 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:30.796979 (+ 155us) client_negotiation.cc:769] Sending connection context
0619 08:56:30.799228 (+ 2249us) client_negotiation.cc:240] Negotiation successful
0619 08:56:30.799328 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":28,"thread_start_us":58,"threads_started":1}
I20260619 08:56:30.800379 7597 heartbeater.cc:347] Connected to a master server at 127.0.98.60:41911
I20260619 08:56:30.800427 7597 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:30.800488 7597 heartbeater.cc:511] Master 127.0.98.60:41911 requested a full tablet report, sending...
I20260619 08:56:30.800493 7599 heartbeater.cc:347] Connected to a master server at 127.0.98.61:46667
I20260619 08:56:30.800539 7599 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:30.800571 7599 heartbeater.cc:511] Master 127.0.98.61:46667 requested a full tablet report, sending...
I20260619 08:56:30.800571 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39819
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39107
--webserver_port=35529
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:39107,127.0.98.61:46667
--tserver_master_addrs=127.0.98.62:39107,127.0.98.61:46667,127.0.98.60:41911 with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 08:56:30.800758 7273 ts_manager.cc:194] Registered new tserver with Master: 14f33dcab292446f9ee74b2a2a56f56d (127.0.98.2:42913)
I20260619 08:56:30.800824 7094 ts_manager.cc:194] Registered new tserver with Master: 14f33dcab292446f9ee74b2a2a56f56d (127.0.98.2:42913)
W20260619 08:56:30.879341 7604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:30.879534 7604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:30.879555 7604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:30.880645 7604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:30.880679 7604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:30.880703 7604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 08:56:30.882081 7604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39819
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35529
--tserver_master_addrs=127.0.98.62:39107,127.0.98.61:46667,127.0.98.60:41911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:30.882272 7604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:30.882443 7604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:30.887096 7612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:30.887082 7610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:30.887110 7609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:30.887348 7604 server_base.cc:1034] running on GCE node
I20260619 08:56:30.887565 7604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:30.887773 7604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:30.888907 7604 hybrid_clock.cc:648] HybridClock initialized: now 1781859390888890 us; error 27 us; skew 500 ppm
Jun 19 08:56:30 dist-test-slave-h4g5 krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859390, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:30.891204 7604 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 08:56:30.891600 7604 webserver.cc:460] Webserver started at http://127.0.98.3:35529/ using document root <none> and password file <none>
I20260619 08:56:30.891795 7604 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:30.891856 7604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:30.893123 7604 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:30.893620 7619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:30.893775 7604 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:30.893852 7604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-2/wal
uuid: "c2237f88ed954837b849c5727176aac3"
format_stamp: "Formatted at 2026-06-19 08:56:29 on dist-test-slave-h4g5"
server_key: "238278d48b8e2765f936978e9b693386"
server_key_iv: "3c93ceccf1855e5baaf00c43c18da35d"
server_key_version: "encryptionkey@0"
I20260619 08:56:30.894174 7604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:30.929996 7604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:30.932137 7604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:30.932279 7604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:30.932502 7604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:30.932793 7604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:30.932825 7604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:30.932853 7604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:30.932899 7604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:30.938175 7604 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39819
I20260619 08:56:30.938230 7732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39819 every 8 connection(s)
I20260619 08:56:30.938548 7604 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 08:56:30 dist-test-slave-h4g5 krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859390, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 08:56:30.945302 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 7604
Jun 19 08:56:30 dist-test-slave-h4g5 krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859390, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 08:56:30 dist-test-slave-h4g5 krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859390, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 08:56:30.950379 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.946298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41911 (local address 127.0.0.1:40998)
0619 08:56:30.946398 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:30.946407 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:30.946445 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:30.947049 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:30.947051 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:30.947053 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:30.947098 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.947100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.947985 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.947986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.949154 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.949157 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.950177 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.950179 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.950237 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.950241 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:56:30.950268 (+ 27us) client_negotiation.cc:240] Negotiation successful
0619 08:56:30.950309 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 08:56:30.950680 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.946514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40998 (local address 127.0.98.60:41911)
0619 08:56:30.946919 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:30.946921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:30.946928 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:56:30.946949 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:30.946950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:30.946973 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:30.947044 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:30.947165 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.947872 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.949264 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.950100 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.950489 (+ 389us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.950566 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:30.950585 (+ 19us) server_negotiation.cc:299] Negotiation successful
0619 08:56:30.950620 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":62,"threads_started":1}
I20260619 08:56:30.950699 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.939544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.98.3:34035)
0619 08:56:30.939673 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:30.939682 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:30.940433 (+ 751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:30.940650 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:30.940654 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:30.941011 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:30.941198 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.941202 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.942151 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.942151 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.942854 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.942856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.943312 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.943313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.943447 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.944127 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:30.944130 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:30.946102 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:30.948169 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.948175 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.948177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.948726 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.949731 (+ 1005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.949732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.949732 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.949783 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.950032 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:30.950039 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:30.950198 (+ 159us) client_negotiation.cc:769] Sending connection context
0619 08:56:30.950298 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 08:56:30.950476 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":41,"thread_start_us":63,"threads_started":1}
I20260619 08:56:30.950943 7291 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 08:56:30.951109 7323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebd9ae8efff44fb09d1537d5c4147426. Latest consensus state: current_term: 1 leader_uuid: "ebd9ae8efff44fb09d1537d5c4147426" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: false } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: true } } }
I20260619 08:56:30.951228 7323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [sys.catalog]: This master's current role is: LEARNER
I20260619 08:56:30.951438 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.939207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46667 (local address 127.0.98.3:35359)
0619 08:56:30.939366 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:30.939414 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:30.940424 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:30.940651 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:30.940655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:30.941023 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:30.941198 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.941204 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.941866 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.941867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.942886 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.942889 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.943355 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.943356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.943444 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.944025 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:30.944034 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:30.945625 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:30.948243 (+ 2618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.948244 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.948245 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.948943 (+ 698us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.949352 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.949352 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.949353 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.949531 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.951135 (+ 1604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:30.951135 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:30.951177 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 08:56:30.951232 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 08:56:30.951325 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":8,"thread_start_us":86,"threads_started":1}
I20260619 08:56:30.951471 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.939181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35359 (local address 127.0.98.61:46667)
0619 08:56:30.939237 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:30.939239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:30.939503 (+ 264us) server_negotiation.cc:407] Connection header received
0619 08:56:30.940545 (+ 1042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:30.940546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:30.940558 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:30.940588 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:30.941300 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.941778 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.942957 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.943242 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.943293 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.945713 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:30.945715 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:30.945715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:30.945722 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:30.947832 (+ 2110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.949218 (+ 1386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.949219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.949220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.949280 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.950917 (+ 1637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.950919 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.950919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.951052 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:30.951107 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:30.951323 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 08:56:30.951394 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 08:56:30.951714 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.939729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34035 (local address 127.0.98.62:39107)
0619 08:56:30.939796 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:30.939798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:30.939806 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:56:30.940519 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:30.940520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:30.940532 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:30.940576 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:30.941280 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.942046 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.942932 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.943230 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.943290 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.946203 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:30.946204 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:30.946205 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:30.946212 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:30.948060 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.949607 (+ 1547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.949608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.949609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.949667 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.949839 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.949840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.949840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.949956 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:30.950010 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:30.951551 (+ 1541us) server_negotiation.cc:299] Negotiation successful
0619 08:56:30.951627 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 08:56:30.952382 7735 heartbeater.cc:347] Connected to a master server at 127.0.98.61:46667
I20260619 08:56:30.952476 7735 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:30.952474 7734 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39107
I20260619 08:56:30.952529 7734 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:30.952672 7734 heartbeater.cc:511] Master 127.0.98.62:39107 requested a full tablet report, sending...
I20260619 08:56:30.952672 7735 heartbeater.cc:511] Master 127.0.98.61:46667 requested a full tablet report, sending...
I20260619 08:56:30.953045 6496 ts_manager.cc:194] Re-registered known tserver with Master: c2237f88ed954837b849c5727176aac3 (127.0.98.3:39819)
I20260619 08:56:30.953140 7094 ts_manager.cc:194] Registered new tserver with Master: c2237f88ed954837b849c5727176aac3 (127.0.98.3:39819)
I20260619 08:56:30.953474 6496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34035
I20260619 08:56:30.954694 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.939237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42497 (local address 127.0.98.60:41911)
0619 08:56:30.939276 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:30.939278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:30.939530 (+ 252us) server_negotiation.cc:407] Connection header received
0619 08:56:30.940544 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:30.940546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:30.940558 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:30.940588 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:30.941308 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.942046 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.943091 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.943293 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.943350 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.946763 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:30.946765 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:30.946765 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:30.946772 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:30.949032 (+ 2260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.953950 (+ 4918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.953951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.953951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.954012 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.954209 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.954209 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.954210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.954316 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:30.954371 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:30.954534 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 08:56:30.954616 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 08:56:30.955174 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.939207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41911 (local address 127.0.98.3:42497)
0619 08:56:30.939367 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:30.939414 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:30.940432 (+ 1018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:30.940651 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:30.940655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:30.941017 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:30.941198 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.941204 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.942124 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.942124 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.943017 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.943018 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.943410 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.943411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.943462 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.944047 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:30.944050 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:30.946676 (+ 2626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:30.953488 (+ 6812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.953489 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.953489 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.953864 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.954078 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:30.954079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:30.954079 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:30.954132 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:30.954395 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:30.954396 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:30.954436 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 08:56:30.954964 (+ 528us) client_negotiation.cc:240] Negotiation successful
0619 08:56:30.955069 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":59,"spinlock_wait_cycles":6912,"thread_start_us":106,"threads_started":1}
I20260619 08:56:30.955587 7733 heartbeater.cc:347] Connected to a master server at 127.0.98.60:41911
I20260619 08:56:30.955641 7733 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:30.955721 7733 heartbeater.cc:511] Master 127.0.98.60:41911 requested a full tablet report, sending...
I20260619 08:56:30.956007 7273 ts_manager.cc:194] Registered new tserver with Master: c2237f88ed954837b849c5727176aac3 (127.0.98.3:39819)
I20260619 08:56:30.957106 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.947603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33330 (local address 127.0.98.62:39107)
0619 08:56:30.948138 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:30.948140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:30.948339 (+ 199us) server_negotiation.cc:407] Connection header received
0619 08:56:30.948358 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:30.948359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:30.948368 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:30.948421 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:30.949817 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.950504 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.951570 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.951863 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.951915 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.953258 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:30.953259 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:30.953259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:30.953265 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:30.954894 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.956006 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.956007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.956008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.956066 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.956555 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.956556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.956557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.956689 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:30.956783 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:30.956932 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 08:56:30.957009 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":72,"threads_started":1}
I20260619 08:56:30.958988 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.947105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41012 (local address 127.0.98.60:41911)
0619 08:56:30.947195 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:30.947196 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:30.948575 (+ 1379us) server_negotiation.cc:407] Connection header received
0619 08:56:30.948595 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:30.948596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:30.948607 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:30.948657 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:30.949417 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.950136 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.954404 (+ 4268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.954700 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.954757 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.956248 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:30.956249 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:30.956250 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:30.956256 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:30.957713 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.958269 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.958270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.958270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.958361 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.958559 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.958560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.958560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.958661 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:30.958728 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:30.958863 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 08:56:30.958927 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":50,"threads_started":1}
I20260619 08:56:30.958639 6496 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:56:30.960525 6496 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:56:30.960548 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.949122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55096 (local address 127.0.98.61:46667)
0619 08:56:30.951531 (+ 2409us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:30.951533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:30.951539 (+ 6us) server_negotiation.cc:407] Connection header received
0619 08:56:30.951556 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:30.951556 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:30.951565 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:30.951615 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:30.952704 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.953394 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.954346 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.954617 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.954675 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.955940 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:30.955942 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:30.955942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:30.955949 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:30.957592 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.958060 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.958061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.958062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.958119 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:30.958285 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:30.958287 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:30.958287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:30.958426 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:30.960172 (+ 1746us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:30.960355 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 08:56:30.960461 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2396,"thread_start_us":50,"threads_started":1}
I20260619 08:56:30.961218 7056 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ebd9ae8efff44fb09d1537d5c4147426" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: false } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: true } } }
I20260619 08:56:30.961304 7056 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:30.962383 7056 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426: attempting to promote NON_VOTER 60b4147da50c4ac78931f669afe3674e to VOTER
I20260619 08:56:30.962509 7056 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: false } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: false } }
I20260619 08:56:30.962865 7104 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [term 1 FOLLOWER]: Refusing update from remote peer ebd9ae8efff44fb09d1537d5c4147426: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 08:56:30.962949 7291 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [term 1 LEARNER]: Refusing update from remote peer ebd9ae8efff44fb09d1537d5c4147426: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 08:56:30.963028 7245 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 08:56:30.963132 7245 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 08:56:30.963680 7244 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 60b4147da50c4ac78931f669afe3674e (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: false } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: false } } }
I20260619 08:56:30.963873 7291 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 60b4147da50c4ac78931f669afe3674e (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: false } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: false } } }
I20260619 08:56:30.964475 7244 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ebd9ae8efff44fb09d1537d5c4147426" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: false } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: false } } }
I20260619 08:56:30.964548 7244 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:30.964593 7244 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ebd9ae8efff44fb09d1537d5c4147426" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: false } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: false } } }
I20260619 08:56:30.964628 7055 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ebd9ae8efff44fb09d1537d5c4147426" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: false } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: false } } }
I20260619 08:56:30.964689 7323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ebd9ae8efff44fb09d1537d5c4147426" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: false } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: false } } }
I20260619 08:56:30.964730 7055 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:30.964756 7323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 08:56:30.964633 7244 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:30.964880 7104 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 60b4147da50c4ac78931f669afe3674e (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: false } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: false } } }
I20260619 08:56:30.965904 7243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ebd9ae8efff44fb09d1537d5c4147426" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: false } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: false } } }
I20260619 08:56:30.965991 7243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 08:56:30.969349 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.965635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39819 (local address 127.0.0.1:43730)
0619 08:56:30.965977 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:30.965986 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:30.966039 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:30.966285 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:30.966286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:30.966288 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:30.966350 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.966352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.967542 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.967543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.968497 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.968499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.969119 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.969120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.969180 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.969186 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 08:56:30.969238 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 08:56:30.969277 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":39,"thread_start_us":70,"threads_started":1}
I20260619 08:56:30.969362 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.965736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43730 (local address 127.0.98.3:39819)
0619 08:56:30.965924 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:30.965928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:30.966090 (+ 162us) server_negotiation.cc:407] Connection header received
0619 08:56:30.966134 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:30.966137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:30.966152 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:30.966224 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:30.966425 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.967439 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.968569 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.969044 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.969106 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.969183 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:30.969253 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 08:56:30.969290 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":120,"threads_started":1}
I20260619 08:56:30.970011 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.965596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42913 (local address 127.0.0.1:55120)
0619 08:56:30.965709 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:30.965719 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:30.965755 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:30.966622 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:30.966623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:30.966624 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:30.966659 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.966660 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.967866 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.967866 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.968926 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.968928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.969821 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.969822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.969881 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.969884 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:56:30.969923 (+ 39us) client_negotiation.cc:240] Negotiation successful
0619 08:56:30.969956 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260619 08:56:30.970180 7667 tablet_service.cc:1467] Processing CreateTablet for tablet c22fad7f34fa4c888996623da0ba9864 (DEFAULT_TABLE table=test-table [id=a95b69d7a8b146c096935b8dc578ba7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:30.970293 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.965837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55120 (local address 127.0.98.2:42913)
0619 08:56:30.966425 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:30.966429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:30.966488 (+ 59us) server_negotiation.cc:407] Connection header received
0619 08:56:30.966527 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:30.966529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:30.966543 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:30.966614 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:30.966736 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.967789 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.969009 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.969742 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.969805 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.970146 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:30.970172 (+ 26us) server_negotiation.cc:299] Negotiation successful
0619 08:56:30.970213 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":65,"threads_started":1}
I20260619 08:56:30.970467 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.965778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37349 (local address 127.0.0.1:36500)
0619 08:56:30.966729 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:30.966738 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:30.966781 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:30.966933 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:30.966934 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:30.966935 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:30.966970 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.966971 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.967731 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.967732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.969075 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:30.969077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.970159 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.970160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:30.970217 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.970220 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:56:30.970256 (+ 36us) client_negotiation.cc:240] Negotiation successful
0619 08:56:30.970291 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":937,"thread_start_us":78,"threads_started":1}
I20260619 08:56:30.970474 7667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c22fad7f34fa4c888996623da0ba9864. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:30.970723 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:30.966140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36500 (local address 127.0.98.1:37349)
0619 08:56:30.966480 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:30.966484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:30.966778 (+ 294us) server_negotiation.cc:407] Connection header received
0619 08:56:30.966847 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:30.966850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:30.966864 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:30.966938 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:30.967018 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.967660 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.969150 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:30.969962 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:30.970033 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:30.970576 (+ 543us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:30.970625 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 08:56:30.970666 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":78,"threads_started":1}
I20260619 08:56:30.970913 7531 tablet_service.cc:1467] Processing CreateTablet for tablet c22fad7f34fa4c888996623da0ba9864 (DEFAULT_TABLE table=test-table [id=a95b69d7a8b146c096935b8dc578ba7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:30.971210 7394 tablet_service.cc:1467] Processing CreateTablet for tablet c22fad7f34fa4c888996623da0ba9864 (DEFAULT_TABLE table=test-table [id=a95b69d7a8b146c096935b8dc578ba7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:30.971460 7394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c22fad7f34fa4c888996623da0ba9864. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:30.971199 7531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c22fad7f34fa4c888996623da0ba9864. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:30.974043 7763 tablet_bootstrap.cc:492] T c22fad7f34fa4c888996623da0ba9864 P c2237f88ed954837b849c5727176aac3: Bootstrap starting.
I20260619 08:56:30.974184 7764 tablet_bootstrap.cc:492] T c22fad7f34fa4c888996623da0ba9864 P b314603327ad4e21bf7d82da8a17eaa3: Bootstrap starting.
I20260619 08:56:30.974648 7763 tablet_bootstrap.cc:654] T c22fad7f34fa4c888996623da0ba9864 P c2237f88ed954837b849c5727176aac3: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:30.974753 7764 tablet_bootstrap.cc:654] T c22fad7f34fa4c888996623da0ba9864 P b314603327ad4e21bf7d82da8a17eaa3: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:30.974941 7763 log.cc:824] T c22fad7f34fa4c888996623da0ba9864 P c2237f88ed954837b849c5727176aac3: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:30.975060 7764 log.cc:824] T c22fad7f34fa4c888996623da0ba9864 P b314603327ad4e21bf7d82da8a17eaa3: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:30.974969 7765 tablet_bootstrap.cc:492] T c22fad7f34fa4c888996623da0ba9864 P 14f33dcab292446f9ee74b2a2a56f56d: Bootstrap starting.
I20260619 08:56:30.975607 7765 tablet_bootstrap.cc:654] T c22fad7f34fa4c888996623da0ba9864 P 14f33dcab292446f9ee74b2a2a56f56d: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:30.975719 7764 tablet_bootstrap.cc:492] T c22fad7f34fa4c888996623da0ba9864 P b314603327ad4e21bf7d82da8a17eaa3: No bootstrap required, opened a new log
I20260619 08:56:30.975776 7764 ts_tablet_manager.cc:1397] T c22fad7f34fa4c888996623da0ba9864 P b314603327ad4e21bf7d82da8a17eaa3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:56:30.975879 7765 log.cc:824] T c22fad7f34fa4c888996623da0ba9864 P 14f33dcab292446f9ee74b2a2a56f56d: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:30.976608 7763 tablet_bootstrap.cc:492] T c22fad7f34fa4c888996623da0ba9864 P c2237f88ed954837b849c5727176aac3: No bootstrap required, opened a new log
I20260619 08:56:30.976619 7765 tablet_bootstrap.cc:492] T c22fad7f34fa4c888996623da0ba9864 P 14f33dcab292446f9ee74b2a2a56f56d: No bootstrap required, opened a new log
I20260619 08:56:30.976670 7763 ts_tablet_manager.cc:1397] T c22fad7f34fa4c888996623da0ba9864 P c2237f88ed954837b849c5727176aac3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 08:56:30.976672 7765 ts_tablet_manager.cc:1397] T c22fad7f34fa4c888996623da0ba9864 P 14f33dcab292446f9ee74b2a2a56f56d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:56:30.978246 7764 raft_consensus.cc:348] T c22fad7f34fa4c888996623da0ba9864 P b314603327ad4e21bf7d82da8a17eaa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f33dcab292446f9ee74b2a2a56f56d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42913 } } peers { permanent_uuid: "c2237f88ed954837b849c5727176aac3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39819 } } peers { permanent_uuid: "b314603327ad4e21bf7d82da8a17eaa3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37349 } }
I20260619 08:56:30.978415 7764 raft_consensus.cc:374] T c22fad7f34fa4c888996623da0ba9864 P b314603327ad4e21bf7d82da8a17eaa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:30.978451 7764 raft_consensus.cc:729] T c22fad7f34fa4c888996623da0ba9864 P b314603327ad4e21bf7d82da8a17eaa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b314603327ad4e21bf7d82da8a17eaa3, State: Initialized, Role: FOLLOWER
I20260619 08:56:30.978485 7763 raft_consensus.cc:348] T c22fad7f34fa4c888996623da0ba9864 P c2237f88ed954837b849c5727176aac3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f33dcab292446f9ee74b2a2a56f56d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42913 } } peers { permanent_uuid: "c2237f88ed954837b849c5727176aac3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39819 } } peers { permanent_uuid: "b314603327ad4e21bf7d82da8a17eaa3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37349 } }
I20260619 08:56:30.978562 7764 consensus_queue.cc:260] T c22fad7f34fa4c888996623da0ba9864 P b314603327ad4e21bf7d82da8a17eaa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f33dcab292446f9ee74b2a2a56f56d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42913 } } peers { permanent_uuid: "c2237f88ed954837b849c5727176aac3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39819 } } peers { permanent_uuid: "b314603327ad4e21bf7d82da8a17eaa3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37349 } }
I20260619 08:56:30.978641 7763 raft_consensus.cc:374] T c22fad7f34fa4c888996623da0ba9864 P c2237f88ed954837b849c5727176aac3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:30.978680 7763 raft_consensus.cc:729] T c22fad7f34fa4c888996623da0ba9864 P c2237f88ed954837b849c5727176aac3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2237f88ed954837b849c5727176aac3, State: Initialized, Role: FOLLOWER
I20260619 08:56:30.978780 7763 consensus_queue.cc:260] T c22fad7f34fa4c888996623da0ba9864 P c2237f88ed954837b849c5727176aac3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f33dcab292446f9ee74b2a2a56f56d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42913 } } peers { permanent_uuid: "c2237f88ed954837b849c5727176aac3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39819 } } peers { permanent_uuid: "b314603327ad4e21bf7d82da8a17eaa3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37349 } }
I20260619 08:56:30.978818 7764 ts_tablet_manager.cc:1428] T c22fad7f34fa4c888996623da0ba9864 P b314603327ad4e21bf7d82da8a17eaa3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:56:30.978938 7461 heartbeater.cc:503] Master 127.0.98.62:39107 was elected leader, sending a full tablet report...
I20260619 08:56:30.979013 7763 ts_tablet_manager.cc:1428] T c22fad7f34fa4c888996623da0ba9864 P c2237f88ed954837b849c5727176aac3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 08:56:30.979178 7734 heartbeater.cc:503] Master 127.0.98.62:39107 was elected leader, sending a full tablet report...
I20260619 08:56:30.978857 7765 raft_consensus.cc:348] T c22fad7f34fa4c888996623da0ba9864 P 14f33dcab292446f9ee74b2a2a56f56d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f33dcab292446f9ee74b2a2a56f56d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42913 } } peers { permanent_uuid: "c2237f88ed954837b849c5727176aac3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39819 } } peers { permanent_uuid: "b314603327ad4e21bf7d82da8a17eaa3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37349 } }
I20260619 08:56:30.979224 7765 raft_consensus.cc:374] T c22fad7f34fa4c888996623da0ba9864 P 14f33dcab292446f9ee74b2a2a56f56d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:30.979264 7765 raft_consensus.cc:729] T c22fad7f34fa4c888996623da0ba9864 P 14f33dcab292446f9ee74b2a2a56f56d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14f33dcab292446f9ee74b2a2a56f56d, State: Initialized, Role: FOLLOWER
I20260619 08:56:30.979375 7765 consensus_queue.cc:260] T c22fad7f34fa4c888996623da0ba9864 P 14f33dcab292446f9ee74b2a2a56f56d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f33dcab292446f9ee74b2a2a56f56d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42913 } } peers { permanent_uuid: "c2237f88ed954837b849c5727176aac3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39819 } } peers { permanent_uuid: "b314603327ad4e21bf7d82da8a17eaa3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37349 } }
I20260619 08:56:30.979609 7765 ts_tablet_manager.cc:1428] T c22fad7f34fa4c888996623da0ba9864 P 14f33dcab292446f9ee74b2a2a56f56d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:56:30.979705 7598 heartbeater.cc:503] Master 127.0.98.62:39107 was elected leader, sending a full tablet report...
W20260619 08:56:31.036824 7600 tablet.cc:2369] T c22fad7f34fa4c888996623da0ba9864 P 14f33dcab292446f9ee74b2a2a56f56d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 08:56:31.146752 7463 tablet.cc:2369] T c22fad7f34fa4c888996623da0ba9864 P b314603327ad4e21bf7d82da8a17eaa3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:56:31.178629 7769 raft_consensus.cc:482] T c22fad7f34fa4c888996623da0ba9864 P b314603327ad4e21bf7d82da8a17eaa3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:56:31.178794 7769 raft_consensus.cc:504] T c22fad7f34fa4c888996623da0ba9864 P b314603327ad4e21bf7d82da8a17eaa3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f33dcab292446f9ee74b2a2a56f56d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42913 } } peers { permanent_uuid: "c2237f88ed954837b849c5727176aac3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39819 } } peers { permanent_uuid: "b314603327ad4e21bf7d82da8a17eaa3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37349 } }
I20260619 08:56:31.179164 7769 leader_election.cc:288] T c22fad7f34fa4c888996623da0ba9864 P b314603327ad4e21bf7d82da8a17eaa3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14f33dcab292446f9ee74b2a2a56f56d (127.0.98.2:42913), c2237f88ed954837b849c5727176aac3 (127.0.98.3:39819)
I20260619 08:56:31.182024 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:31.179460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36575 (local address 127.0.98.3:39819)
0619 08:56:31.179504 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:31.179506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:31.179518 (+ 12us) server_negotiation.cc:407] Connection header received
0619 08:56:31.179704 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:31.179705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:31.179714 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:31.179760 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:31.179922 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:31.180443 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:31.181379 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:31.181723 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:31.181780 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:31.181791 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:31.181922 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 08:56:31.181952 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 08:56:31.182068 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:31.179350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39819 (local address 127.0.98.1:36575)
0619 08:56:31.179468 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:31.179478 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:31.179535 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:31.179775 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:31.179776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:31.179777 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:31.179822 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:31.179824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:31.180512 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:31.180514 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:31.181299 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:31.181301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:31.181795 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:31.181796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:31.181865 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:31.181869 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:56:31.181912 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 08:56:31.181951 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":35,"thread_start_us":50,"threads_started":1}
I20260619 08:56:31.182080 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:31.179426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33627 (local address 127.0.98.2:42913)
0619 08:56:31.179473 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:31.179475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:31.179542 (+ 67us) server_negotiation.cc:407] Connection header received
0619 08:56:31.179626 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:31.179627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:31.179640 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:31.179695 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:31.179844 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:31.180366 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:31.181389 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:31.181743 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:31.181817 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:31.181828 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:31.181964 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 08:56:31.182009 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 08:56:31.182297 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:31.179331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42913 (local address 127.0.98.1:33627)
0619 08:56:31.179451 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:31.179465 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:31.179553 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:31.179703 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:31.179704 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:31.179706 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:31.179766 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:31.179768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:31.180448 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:31.180449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:31.181294 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:31.181297 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:31.181834 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:31.181835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:31.181898 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:31.181901 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:56:31.181953 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 08:56:31.182004 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":32,"thread_start_us":65,"threads_started":1}
I20260619 08:56:31.182647 7687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c22fad7f34fa4c888996623da0ba9864" candidate_uuid: "b314603327ad4e21bf7d82da8a17eaa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2237f88ed954837b849c5727176aac3" is_pre_election: true
I20260619 08:56:31.182648 7551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c22fad7f34fa4c888996623da0ba9864" candidate_uuid: "b314603327ad4e21bf7d82da8a17eaa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14f33dcab292446f9ee74b2a2a56f56d" is_pre_election: true
I20260619 08:56:31.182782 7551 raft_consensus.cc:2454] T c22fad7f34fa4c888996623da0ba9864 P 14f33dcab292446f9ee74b2a2a56f56d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b314603327ad4e21bf7d82da8a17eaa3 in term 0.
I20260619 08:56:31.182785 7687 raft_consensus.cc:2454] T c22fad7f34fa4c888996623da0ba9864 P c2237f88ed954837b849c5727176aac3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b314603327ad4e21bf7d82da8a17eaa3 in term 0.
I20260619 08:56:31.183009 7347 leader_election.cc:302] T c22fad7f34fa4c888996623da0ba9864 P b314603327ad4e21bf7d82da8a17eaa3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b314603327ad4e21bf7d82da8a17eaa3, c2237f88ed954837b849c5727176aac3; no voters:
I20260619 08:56:31.183141 7769 raft_consensus.cc:2781] T c22fad7f34fa4c888996623da0ba9864 P b314603327ad4e21bf7d82da8a17eaa3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:56:31.183190 7769 raft_consensus.cc:482] T c22fad7f34fa4c888996623da0ba9864 P b314603327ad4e21bf7d82da8a17eaa3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:56:31.183204 7769 raft_consensus.cc:3037] T c22fad7f34fa4c888996623da0ba9864 P b314603327ad4e21bf7d82da8a17eaa3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:31.183823 7769 raft_consensus.cc:504] T c22fad7f34fa4c888996623da0ba9864 P b314603327ad4e21bf7d82da8a17eaa3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f33dcab292446f9ee74b2a2a56f56d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42913 } } peers { permanent_uuid: "c2237f88ed954837b849c5727176aac3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39819 } } peers { permanent_uuid: "b314603327ad4e21bf7d82da8a17eaa3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37349 } }
I20260619 08:56:31.183947 7769 leader_election.cc:288] T c22fad7f34fa4c888996623da0ba9864 P b314603327ad4e21bf7d82da8a17eaa3 [CANDIDATE]: Term 1 election: Requested vote from peers 14f33dcab292446f9ee74b2a2a56f56d (127.0.98.2:42913), c2237f88ed954837b849c5727176aac3 (127.0.98.3:39819)
I20260619 08:56:31.184116 7551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c22fad7f34fa4c888996623da0ba9864" candidate_uuid: "b314603327ad4e21bf7d82da8a17eaa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14f33dcab292446f9ee74b2a2a56f56d"
I20260619 08:56:31.184172 7687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c22fad7f34fa4c888996623da0ba9864" candidate_uuid: "b314603327ad4e21bf7d82da8a17eaa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2237f88ed954837b849c5727176aac3"
I20260619 08:56:31.184208 7551 raft_consensus.cc:3037] T c22fad7f34fa4c888996623da0ba9864 P 14f33dcab292446f9ee74b2a2a56f56d [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:31.184252 7687 raft_consensus.cc:3037] T c22fad7f34fa4c888996623da0ba9864 P c2237f88ed954837b849c5727176aac3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:31.184877 7551 raft_consensus.cc:2454] T c22fad7f34fa4c888996623da0ba9864 P 14f33dcab292446f9ee74b2a2a56f56d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b314603327ad4e21bf7d82da8a17eaa3 in term 1.
I20260619 08:56:31.184923 7687 raft_consensus.cc:2454] T c22fad7f34fa4c888996623da0ba9864 P c2237f88ed954837b849c5727176aac3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b314603327ad4e21bf7d82da8a17eaa3 in term 1.
I20260619 08:56:31.185045 7350 leader_election.cc:302] T c22fad7f34fa4c888996623da0ba9864 P b314603327ad4e21bf7d82da8a17eaa3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14f33dcab292446f9ee74b2a2a56f56d, b314603327ad4e21bf7d82da8a17eaa3; no voters:
I20260619 08:56:31.185165 7769 raft_consensus.cc:2781] T c22fad7f34fa4c888996623da0ba9864 P b314603327ad4e21bf7d82da8a17eaa3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:31.185271 7769 raft_consensus.cc:686] T c22fad7f34fa4c888996623da0ba9864 P b314603327ad4e21bf7d82da8a17eaa3 [term 1 LEADER]: Becoming Leader. State: Replica: b314603327ad4e21bf7d82da8a17eaa3, State: Running, Role: LEADER
I20260619 08:56:31.185413 7769 consensus_queue.cc:237] T c22fad7f34fa4c888996623da0ba9864 P b314603327ad4e21bf7d82da8a17eaa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f33dcab292446f9ee74b2a2a56f56d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42913 } } peers { permanent_uuid: "c2237f88ed954837b849c5727176aac3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39819 } } peers { permanent_uuid: "b314603327ad4e21bf7d82da8a17eaa3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37349 } }
I20260619 08:56:31.186131 6496 catalog_manager.cc:5495] T c22fad7f34fa4c888996623da0ba9864 P b314603327ad4e21bf7d82da8a17eaa3 reported cstate change: term changed from 0 to 1, leader changed from <none> to b314603327ad4e21bf7d82da8a17eaa3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b314603327ad4e21bf7d82da8a17eaa3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f33dcab292446f9ee74b2a2a56f56d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2237f88ed954837b849c5727176aac3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b314603327ad4e21bf7d82da8a17eaa3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37349 } health_report { overall_health: HEALTHY } } }
W20260619 08:56:31.189157 7736 tablet.cc:2369] T c22fad7f34fa4c888996623da0ba9864 P c2237f88ed954837b849c5727176aac3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:56:31.194211 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:31.192024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36514 (local address 127.0.98.1:37349)
0619 08:56:31.192117 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:31.192119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:31.192129 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:56:31.192159 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:31.192160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:31.192176 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:31.192233 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:56:31.192358 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:31.192791 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:31.193503 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:31.193683 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:31.193730 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:31.193891 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:56:31.193981 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:56:31.194042 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:31.194139 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 08:56:31.194155 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 08:56:31.195804 7687 raft_consensus.cc:1261] T c22fad7f34fa4c888996623da0ba9864 P c2237f88ed954837b849c5727176aac3 [term 1 FOLLOWER]: Refusing update from remote peer b314603327ad4e21bf7d82da8a17eaa3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:56:31.196146 7551 raft_consensus.cc:1261] T c22fad7f34fa4c888996623da0ba9864 P 14f33dcab292446f9ee74b2a2a56f56d [term 1 FOLLOWER]: Refusing update from remote peer b314603327ad4e21bf7d82da8a17eaa3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:56:31.196195 7774 consensus_queue.cc:1035] T c22fad7f34fa4c888996623da0ba9864 P b314603327ad4e21bf7d82da8a17eaa3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2237f88ed954837b849c5727176aac3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:56:31.196379 7769 consensus_queue.cc:1035] T c22fad7f34fa4c888996623da0ba9864 P b314603327ad4e21bf7d82da8a17eaa3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14f33dcab292446f9ee74b2a2a56f56d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:56:31.199076 7779 mvcc.cc:205] Tried to move back new op lower bound from 7298496066335842304 to 7298496066295955456. Current Snapshot: MvccSnapshot[applied={T|T < 7298496066335842304}]
I20260619 08:56:31.199872 7778 mvcc.cc:205] Tried to move back new op lower bound from 7298496066335842304 to 7298496066295955456. Current Snapshot: MvccSnapshot[applied={T|T < 7298496066335842304}]
I20260619 08:56:31.201208 6496 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:56:31.201284 6496 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:56:31.202775 6496 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ebd9ae8efff44fb09d1537d5c4147426: Sending DeleteTablet for 3 replicas of tablet c22fad7f34fa4c888996623da0ba9864
I20260619 08:56:31.203047 7531 tablet_service.cc:1514] Processing DeleteTablet for tablet c22fad7f34fa4c888996623da0ba9864 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55120
I20260619 08:56:31.203120 7394 tablet_service.cc:1514] Processing DeleteTablet for tablet c22fad7f34fa4c888996623da0ba9864 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36500
I20260619 08:56:31.203253 7667 tablet_service.cc:1514] Processing DeleteTablet for tablet c22fad7f34fa4c888996623da0ba9864 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43730
I20260619 08:56:31.203456 7788 tablet_replica.cc:330] T c22fad7f34fa4c888996623da0ba9864 P c2237f88ed954837b849c5727176aac3: stopping tablet replica
I20260619 08:56:31.203563 7788 raft_consensus.cc:2229] T c22fad7f34fa4c888996623da0ba9864 P c2237f88ed954837b849c5727176aac3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:56:31.203665 7788 raft_consensus.cc:2258] T c22fad7f34fa4c888996623da0ba9864 P c2237f88ed954837b849c5727176aac3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:31.203661 7786 tablet_replica.cc:330] T c22fad7f34fa4c888996623da0ba9864 P 14f33dcab292446f9ee74b2a2a56f56d: stopping tablet replica
I20260619 08:56:31.203917 7786 raft_consensus.cc:2229] T c22fad7f34fa4c888996623da0ba9864 P 14f33dcab292446f9ee74b2a2a56f56d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:56:31.204059 7786 raft_consensus.cc:2258] T c22fad7f34fa4c888996623da0ba9864 P 14f33dcab292446f9ee74b2a2a56f56d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:31.204137 7788 ts_tablet_manager.cc:1905] T c22fad7f34fa4c888996623da0ba9864 P c2237f88ed954837b849c5727176aac3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:31.204550 7786 ts_tablet_manager.cc:1905] T c22fad7f34fa4c888996623da0ba9864 P 14f33dcab292446f9ee74b2a2a56f56d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:31.204778 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 6464
I20260619 08:56:31.206091 7788 ts_tablet_manager.cc:1918] T c22fad7f34fa4c888996623da0ba9864 P c2237f88ed954837b849c5727176aac3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:56:31.206162 7788 log.cc:1196] T c22fad7f34fa4c888996623da0ba9864 P c2237f88ed954837b849c5727176aac3: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-2/wal/wals/c22fad7f34fa4c888996623da0ba9864
I20260619 08:56:31.206415 7788 ts_tablet_manager.cc:1939] T c22fad7f34fa4c888996623da0ba9864 P c2237f88ed954837b849c5727176aac3: Deleting consensus metadata
I20260619 08:56:31.206977 7786 ts_tablet_manager.cc:1918] T c22fad7f34fa4c888996623da0ba9864 P 14f33dcab292446f9ee74b2a2a56f56d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:56:31.207090 7786 log.cc:1196] T c22fad7f34fa4c888996623da0ba9864 P 14f33dcab292446f9ee74b2a2a56f56d: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-1/wal/wals/c22fad7f34fa4c888996623da0ba9864
I20260619 08:56:31.207362 7786 ts_tablet_manager.cc:1939] T c22fad7f34fa4c888996623da0ba9864 P 14f33dcab292446f9ee74b2a2a56f56d: Deleting consensus metadata
I20260619 08:56:31.208395 7787 tablet_replica.cc:330] T c22fad7f34fa4c888996623da0ba9864 P b314603327ad4e21bf7d82da8a17eaa3: stopping tablet replica
I20260619 08:56:31.208499 7787 raft_consensus.cc:2229] T c22fad7f34fa4c888996623da0ba9864 P b314603327ad4e21bf7d82da8a17eaa3 [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:56:31.208597 7787 raft_consensus.cc:2258] T c22fad7f34fa4c888996623da0ba9864 P b314603327ad4e21bf7d82da8a17eaa3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:31.208947 7787 ts_tablet_manager.cc:1905] T c22fad7f34fa4c888996623da0ba9864 P b314603327ad4e21bf7d82da8a17eaa3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:31.210539 7787 ts_tablet_manager.cc:1918] T c22fad7f34fa4c888996623da0ba9864 P b314603327ad4e21bf7d82da8a17eaa3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260619 08:56:31.210666 7623 connection.cc:535] server connection from 127.0.0.1:43730 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260619 08:56:31.210767 7484 connection.cc:535] server connection from 127.0.0.1:55120 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 08:56:31.210798 7787 log.cc:1196] T c22fad7f34fa4c888996623da0ba9864 P b314603327ad4e21bf7d82da8a17eaa3: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-0/wal/wals/c22fad7f34fa4c888996623da0ba9864
I20260619 08:56:31.211041 7787 ts_tablet_manager.cc:1939] T c22fad7f34fa4c888996623da0ba9864 P b314603327ad4e21bf7d82da8a17eaa3: Deleting consensus metadata
W20260619 08:56:31.211447 7347 connection.cc:441] server connection from 127.0.0.1:36500 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36500 (request call id 1) could send its response
I20260619 08:56:31.216643 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:31.204305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41018 (local address 127.0.98.60:41911)
0619 08:56:31.204351 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:31.204352 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:31.208266 (+ 3914us) server_negotiation.cc:407] Connection header received
0619 08:56:31.208327 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:31.208328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:31.208344 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:31.208396 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:31.209420 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:31.210175 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:31.211400 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:31.211687 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:31.211729 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:31.213649 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:31.213651 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:31.213651 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:31.213658 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:31.215247 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:31.215603 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:31.215604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:31.215604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:31.215658 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:31.215848 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:31.215849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:31.215849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:31.215974 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:31.216060 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:31.216462 (+ 402us) server_negotiation.cc:299] Negotiation successful
0619 08:56:31.216544 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 08:56:31.219805 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:31.212413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55108 (local address 127.0.98.61:46667)
0619 08:56:31.212453 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:31.212454 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:31.212694 (+ 240us) server_negotiation.cc:407] Connection header received
0619 08:56:31.212784 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:31.212785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:31.212808 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:31.212837 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:31.213716 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:31.214395 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:31.215409 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:31.215684 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:31.215722 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:31.217387 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:31.217389 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:31.217389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:31.217396 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:31.218730 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:31.219051 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:31.219052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:31.219052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:31.219110 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:31.219264 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:31.219265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:31.219266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:31.219390 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:31.219482 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:31.219642 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 08:56:31.219715 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 08:56:31.220656 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:31.212411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41026 (local address 127.0.98.60:41911)
0619 08:56:31.212446 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:31.212448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:31.212744 (+ 296us) server_negotiation.cc:407] Connection header received
0619 08:56:31.212847 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:31.212848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:31.212858 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:31.212886 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:31.213526 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:31.214747 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:31.215779 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:31.216350 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:31.216392 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:31.217703 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:31.217704 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:31.217704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:31.217711 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:31.219288 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:31.219576 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:31.219577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:31.219577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:31.219644 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:31.220195 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:31.220196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:31.220196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:31.220304 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:31.220356 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:31.220512 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 08:56:31.220575 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 08:56:32.200698 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:32.200436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.98.1:43645)
0619 08:56:32.200569 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:32.200642 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20260619 08:56:32.200876 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:32.200619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.98.2:45077)
0619 08:56:32.200766 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:32.200834 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":51,"threads_started":1}
W20260619 08:56:32.200978 7461 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39107 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
W20260619 08:56:32.201110 7598 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39107 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
I20260619 08:56:32.201421 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:32.201226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.98.3:35979)
0619 08:56:32.201331 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:32.201383 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":41,"threads_started":1}
W20260619 08:56:32.201596 7734 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39107 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
I20260619 08:56:32.702984 7800 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ebd9ae8efff44fb09d1537d5c4147426)
I20260619 08:56:32.703152 7800 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: false } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: false } }
I20260619 08:56:32.703399 7800 leader_election.cc:288] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ebd9ae8efff44fb09d1537d5c4147426 (127.0.98.62:39107), 60b4147da50c4ac78931f669afe3674e (127.0.98.60:41911)
I20260619 08:56:32.703658 7803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ebd9ae8efff44fb09d1537d5c4147426)
I20260619 08:56:32.703742 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:32.703539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.0.1:48102)
0619 08:56:32.703644 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:32.703695 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":47,"threads_started":1}
I20260619 08:56:32.703779 7803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: false } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: false } }
W20260619 08:56:32.703871 7081 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
I20260619 08:56:32.704033 7803 leader_election.cc:288] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ebd9ae8efff44fb09d1537d5c4147426 (127.0.98.62:39107), 3ff9d07cc16e4b48aff2249b4cdfdf46 (127.0.98.61:46667)
I20260619 08:56:32.704407 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:32.704322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.0.1:48118)
0619 08:56:32.704362 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:32.704387 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260619 08:56:32.704411 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:32.704144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.0.1:48110)
0619 08:56:32.704327 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:32.704367 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":47,"threads_started":1}
W20260619 08:56:32.704480 7081 leader_election.cc:334] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ebd9ae8efff44fb09d1537d5c4147426 (127.0.98.62:39107): Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
W20260619 08:56:32.704636 7268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
I20260619 08:56:32.705010 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:32.704922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.0.1:48132)
0619 08:56:32.704957 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:32.704988 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260619 08:56:32.705931 7268 leader_election.cc:334] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ebd9ae8efff44fb09d1537d5c4147426 (127.0.98.62:39107): Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
I20260619 08:56:32.706676 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:32.703542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41911 (local address 127.0.0.1:34936)
0619 08:56:32.703656 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:32.703667 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:32.703722 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:32.703954 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:32.703955 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:32.703958 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:32.704039 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:32.704041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:32.704997 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:32.704998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:32.705822 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:32.705824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:32.706421 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:32.706422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:32.706506 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:32.706509 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:56:32.706562 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 08:56:32.706607 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":50,"threads_started":1}
I20260619 08:56:32.706674 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:32.703711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34936 (local address 127.0.98.60:41911)
0619 08:56:32.703823 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:32.703825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:32.703833 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:56:32.703858 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:32.703858 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:32.703868 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:32.704147 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:32.704186 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:32.704933 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:32.705902 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:32.706350 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:32.706412 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:32.706423 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:32.706572 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 08:56:32.706612 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":44,"threads_started":1}
I20260619 08:56:32.706888 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:32.704208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46667 (local address 127.0.0.1:33356)
0619 08:56:32.704396 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:32.704406 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:32.704453 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:32.704643 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:32.704644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:32.704645 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:32.704682 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:32.704684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:32.705391 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:32.705392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:32.706159 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:32.706161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:32.706656 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:32.706657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:32.706716 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:32.706720 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:56:32.706768 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 08:56:32.706813 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":36,"threads_started":1}
I20260619 08:56:32.706988 7291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "60b4147da50c4ac78931f669afe3674e" is_pre_election: true
I20260619 08:56:32.707096 7291 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ff9d07cc16e4b48aff2249b4cdfdf46 in term 1.
I20260619 08:56:32.707124 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:32.704273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33356 (local address 127.0.98.61:46667)
0619 08:56:32.704526 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:32.704528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:32.704533 (+ 5us) server_negotiation.cc:407] Connection header received
0619 08:56:32.704551 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:32.704552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:32.704561 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:32.704610 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:32.704739 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:32.705327 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:32.706227 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:32.706591 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:32.706648 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:32.706657 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:32.707023 (+ 366us) server_negotiation.cc:299] Negotiation successful
0619 08:56:32.707067 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":64,"threads_started":1}
I20260619 08:56:32.707244 7078 leader_election.cc:302] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ff9d07cc16e4b48aff2249b4cdfdf46, 60b4147da50c4ac78931f669afe3674e; no voters: ebd9ae8efff44fb09d1537d5c4147426
I20260619 08:56:32.707299 7104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "60b4147da50c4ac78931f669afe3674e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" is_pre_election: true
I20260619 08:56:32.707345 7800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 08:56:32.707391 7104 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b4147da50c4ac78931f669afe3674e in term 1.
I20260619 08:56:32.707432 7800 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ebd9ae8efff44fb09d1537d5c4147426)
I20260619 08:56:32.707481 7800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [term 1 FOLLOWER]: Advancing to term 2
I20260619 08:56:32.707545 7266 leader_election.cc:302] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ff9d07cc16e4b48aff2249b4cdfdf46, 60b4147da50c4ac78931f669afe3674e; no voters: ebd9ae8efff44fb09d1537d5c4147426
I20260619 08:56:32.707634 7803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 08:56:32.707679 7803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [term 1 FOLLOWER]: Starting leader election (detected failure of leader ebd9ae8efff44fb09d1537d5c4147426)
I20260619 08:56:32.707696 7803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [term 1 FOLLOWER]: Advancing to term 2
I20260619 08:56:32.708230 7800 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: false } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: false } }
I20260619 08:56:32.708350 7800 leader_election.cc:288] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [CANDIDATE]: Term 2 election: Requested vote from peers ebd9ae8efff44fb09d1537d5c4147426 (127.0.98.62:39107), 60b4147da50c4ac78931f669afe3674e (127.0.98.60:41911)
I20260619 08:56:32.708395 7803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: false } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: false } }
I20260619 08:56:32.708513 7803 leader_election.cc:288] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [CANDIDATE]: Term 2 election: Requested vote from peers ebd9ae8efff44fb09d1537d5c4147426 (127.0.98.62:39107), 3ff9d07cc16e4b48aff2249b4cdfdf46 (127.0.98.61:46667)
I20260619 08:56:32.708494 7291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "60b4147da50c4ac78931f669afe3674e"
I20260619 08:56:32.708549 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:32.708474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.0.1:48136)
0619 08:56:32.708502 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:32.708527 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260619 08:56:32.708565 7291 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3ff9d07cc16e4b48aff2249b4cdfdf46 in current term 2: Already voted for candidate 60b4147da50c4ac78931f669afe3674e in this term.
I20260619 08:56:32.708691 7104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "60b4147da50c4ac78931f669afe3674e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46"
I20260619 08:56:32.708770 7104 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 60b4147da50c4ac78931f669afe3674e in current term 2: Already voted for candidate 3ff9d07cc16e4b48aff2249b4cdfdf46 in this term.
I20260619 08:56:32.708817 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:32.708727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.0.1:48156)
0619 08:56:32.708756 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:32.708776 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260619 08:56:32.708830 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:32.708625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.0.1:48146)
0619 08:56:32.708730 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:32.708809 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260619 08:56:32.708863 7081 leader_election.cc:334] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ebd9ae8efff44fb09d1537d5c4147426 (127.0.98.62:39107): Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
I20260619 08:56:32.708923 7081 leader_election.cc:302] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3ff9d07cc16e4b48aff2249b4cdfdf46; no voters: 60b4147da50c4ac78931f669afe3674e, ebd9ae8efff44fb09d1537d5c4147426
I20260619 08:56:32.709021 7800 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 08:56:32.709163 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:32.709086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.0.1:48164)
0619 08:56:32.709118 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:32.709145 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260619 08:56:32.709223 7268 leader_election.cc:334] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ebd9ae8efff44fb09d1537d5c4147426 (127.0.98.62:39107): Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
I20260619 08:56:32.709267 7268 leader_election.cc:302] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60b4147da50c4ac78931f669afe3674e; no voters: 3ff9d07cc16e4b48aff2249b4cdfdf46, ebd9ae8efff44fb09d1537d5c4147426
I20260619 08:56:32.709319 7803 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 08:56:33.201563 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:33.201322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.98.1:37639)
0619 08:56:33.201466 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:33.201508 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":95,"threads_started":1}
I20260619 08:56:33.201603 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:33.201409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.98.2:48967)
0619 08:56:33.201523 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:33.201557 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":50,"threads_started":1}
I20260619 08:56:33.202128 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:33.201928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.98.3:50291)
0619 08:56:33.202068 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:33.202097 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":55,"threads_started":1}
I20260619 08:56:34.202280 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:34.202043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.98.1:60503)
0619 08:56:34.202185 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:34.202225 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":64,"threads_started":1}
I20260619 08:56:34.202410 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:34.202203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.98.2:53291)
0619 08:56:34.202317 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:34.202363 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":52,"threads_started":1}
I20260619 08:56:34.202654 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:34.202467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.98.3:54905)
0619 08:56:34.202571 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:34.202605 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":50,"threads_started":1}
I20260619 08:56:34.759156 7816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:56:34.759254 7816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: false } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: false } }
I20260619 08:56:34.759451 7816 leader_election.cc:288] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebd9ae8efff44fb09d1537d5c4147426 (127.0.98.62:39107), 60b4147da50c4ac78931f669afe3674e (127.0.98.60:41911)
I20260619 08:56:34.759689 7291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "60b4147da50c4ac78931f669afe3674e" is_pre_election: true
I20260619 08:56:34.759785 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:34.759592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.0.1:48430)
0619 08:56:34.759707 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:34.759744 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
I20260619 08:56:34.759804 7291 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ff9d07cc16e4b48aff2249b4cdfdf46 in term 2.
I20260619 08:56:34.760030 7078 leader_election.cc:302] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ff9d07cc16e4b48aff2249b4cdfdf46, 60b4147da50c4ac78931f669afe3674e; no voters:
I20260619 08:56:34.760157 7816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260619 08:56:34.760200 7816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:56:34.760218 7816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [term 2 FOLLOWER]: Advancing to term 3
I20260619 08:56:34.760248 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:34.760096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.0.1:48446)
0619 08:56:34.760188 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:34.760214 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260619 08:56:34.760299 7081 leader_election.cc:334] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ebd9ae8efff44fb09d1537d5c4147426 (127.0.98.62:39107): Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
I20260619 08:56:34.760957 7816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: false } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: false } }
I20260619 08:56:34.761071 7816 leader_election.cc:288] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [CANDIDATE]: Term 3 election: Requested vote from peers ebd9ae8efff44fb09d1537d5c4147426 (127.0.98.62:39107), 60b4147da50c4ac78931f669afe3674e (127.0.98.60:41911)
I20260619 08:56:34.761242 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:34.761138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.0.1:48452)
0619 08:56:34.761196 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:34.761225 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 08:56:34.761214 7291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "60b4147da50c4ac78931f669afe3674e"
I20260619 08:56:34.761279 7291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [term 2 FOLLOWER]: Advancing to term 3
I20260619 08:56:34.761538 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:34.761449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.0.1:48462)
0619 08:56:34.761497 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:34.761521 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 08:56:34.761647 7081 leader_election.cc:334] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ebd9ae8efff44fb09d1537d5c4147426 (127.0.98.62:39107): Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
I20260619 08:56:34.761965 7291 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ff9d07cc16e4b48aff2249b4cdfdf46 in term 3.
I20260619 08:56:34.762135 7078 leader_election.cc:302] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3ff9d07cc16e4b48aff2249b4cdfdf46, 60b4147da50c4ac78931f669afe3674e; no voters: ebd9ae8efff44fb09d1537d5c4147426
I20260619 08:56:34.762233 7816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 08:56:34.762348 7816 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [term 3 LEADER]: Becoming Leader. State: Replica: 3ff9d07cc16e4b48aff2249b4cdfdf46, State: Running, Role: LEADER
I20260619 08:56:34.762449 7816 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: false } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: false } }
I20260619 08:56:34.762781 7819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ff9d07cc16e4b48aff2249b4cdfdf46. Latest consensus state: current_term: 3 leader_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: false } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: false } } }
I20260619 08:56:34.762899 7819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:34.763057 7821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:56:34.763468 7821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:56:34.763528 7821 catalog_manager.cc:1259] Loaded cluster ID: 8d8e9b83122a417ea361f872fee64a84
I20260619 08:56:34.763557 7821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:56:34.763742 7821 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:56:34.763814 7821 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46: Loaded TSK: 0
I20260619 08:56:34.763958 7821 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:56:34.769793 7093 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:56:34.771322 7093 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:56:34.772295 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:34.772164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.0.1:48480)
0619 08:56:34.772208 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:34.772272 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 08:56:34.772289 7291 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [term 3 FOLLOWER]: Refusing update from remote peer 3ff9d07cc16e4b48aff2249b4cdfdf46: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260619 08:56:34.772485 7819 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260619 08:56:34.772783 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:34.772689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.0.1:48484)
0619 08:56:34.772730 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:34.772759 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 08:56:34.772855 7081 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 -> Peer ebd9ae8efff44fb09d1537d5c4147426 (127.0.98.62:39107): Couldn't send request to peer ebd9ae8efff44fb09d1537d5c4147426. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 08:56:34.773041 7823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ff9d07cc16e4b48aff2249b4cdfdf46. Latest consensus state: current_term: 3 leader_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: false } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: false } } }
I20260619 08:56:34.773125 7823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 08:56:34.773468 7816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: false } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: false } } }
I20260619 08:56:34.773609 7816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:34.773736 7824 mvcc.cc:205] Tried to move back new op lower bound from 7298496080985219072 to 7298496080947343360. Current Snapshot: MvccSnapshot[applied={T|T < 7298496080985219072}]
I20260619 08:56:34.773810 7823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ebd9ae8efff44fb09d1537d5c4147426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39107 } } peers { permanent_uuid: "3ff9d07cc16e4b48aff2249b4cdfdf46" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46667 } attrs { promote: false } } peers { permanent_uuid: "60b4147da50c4ac78931f669afe3674e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41911 } attrs { promote: false } } }
I20260619 08:56:34.773875 7823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60b4147da50c4ac78931f669afe3674e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 08:56:34.778379 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:34.775125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56136 (local address 127.0.98.2:42913)
0619 08:56:34.775245 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:34.775247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:34.775253 (+ 6us) server_negotiation.cc:407] Connection header received
0619 08:56:34.775280 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:34.775280 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:34.775291 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:34.775494 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:34.775526 (+ 32us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:34.776157 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:34.777496 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:34.778060 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:34.778158 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:34.778171 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:34.778278 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 08:56:34.778312 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":73,"threads_started":1}
I20260619 08:56:34.778378 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:34.775047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42913 (local address 127.0.0.1:56136)
0619 08:56:34.775083 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:34.775091 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:34.775132 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:34.775381 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:34.775382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:34.775384 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:34.775439 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:34.775441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:34.776257 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:34.776258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:34.777411 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:34.777413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:34.778157 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:34.778158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:34.778215 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:34.778218 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:56:34.778262 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 08:56:34.778304 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260619 08:56:34.778602 7531 tablet_service.cc:1467] Processing CreateTablet for tablet b4a08c7d3b7a4ce48b18f79ff41e6ef5 (DEFAULT_TABLE table=test-table [id=f33dbd42cc964795b6fa178c1537781b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:34.778710 7531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4a08c7d3b7a4ce48b18f79ff41e6ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:34.778992 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:34.775284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39819 (local address 127.0.0.1:37908)
0619 08:56:34.776178 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:34.776186 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:34.776259 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:34.776376 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:34.776376 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:34.776378 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:34.776423 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:34.776425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:34.777067 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:34.777068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:34.778067 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:34.778069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:34.778766 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:34.778767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:34.778828 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:34.778832 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:56:34.778894 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 08:56:34.778932 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"thread_start_us":56,"threads_started":1}
I20260619 08:56:34.779023 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:34.775348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37908 (local address 127.0.98.3:39819)
0619 08:56:34.775440 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:34.775442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:34.776234 (+ 792us) server_negotiation.cc:407] Connection header received
0619 08:56:34.776314 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:34.776315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:34.776324 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:34.776376 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:34.776470 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:34.776999 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:34.778133 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:34.778690 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:34.778757 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:34.778768 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:34.778902 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 08:56:34.778942 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":47,"threads_started":1}
I20260619 08:56:34.779309 7667 tablet_service.cc:1467] Processing CreateTablet for tablet b4a08c7d3b7a4ce48b18f79ff41e6ef5 (DEFAULT_TABLE table=test-table [id=f33dbd42cc964795b6fa178c1537781b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:34.779348 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:34.775301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43972 (local address 127.0.98.1:37349)
0619 08:56:34.775429 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:34.775431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:34.775548 (+ 117us) server_negotiation.cc:407] Connection header received
0619 08:56:34.775605 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:34.775606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:34.775616 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:34.775679 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:34.775798 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:34.776493 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:34.777761 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:34.778301 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:34.778361 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:34.778371 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:34.779020 (+ 649us) server_negotiation.cc:299] Negotiation successful
0619 08:56:34.779284 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
I20260619 08:56:34.779428 7667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4a08c7d3b7a4ce48b18f79ff41e6ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:34.779498 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:34.775110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37349 (local address 127.0.0.1:43972)
0619 08:56:34.775485 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:34.775494 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:34.775547 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:34.775679 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:34.775680 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:34.775682 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:34.775726 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:34.775727 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:34.776578 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:34.776578 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:34.777648 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:34.777650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:34.778420 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:34.778421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:34.778888 (+ 467us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:34.778892 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:56:34.778940 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 08:56:34.778979 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":80,"threads_started":1}
I20260619 08:56:34.779709 7394 tablet_service.cc:1467] Processing CreateTablet for tablet b4a08c7d3b7a4ce48b18f79ff41e6ef5 (DEFAULT_TABLE table=test-table [id=f33dbd42cc964795b6fa178c1537781b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:34.779824 7394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4a08c7d3b7a4ce48b18f79ff41e6ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:34.780094 7833 tablet_bootstrap.cc:492] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P 14f33dcab292446f9ee74b2a2a56f56d: Bootstrap starting.
I20260619 08:56:34.780479 7833 tablet_bootstrap.cc:654] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P 14f33dcab292446f9ee74b2a2a56f56d: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:34.780647 7834 tablet_bootstrap.cc:492] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P c2237f88ed954837b849c5727176aac3: Bootstrap starting.
I20260619 08:56:34.780998 7833 tablet_bootstrap.cc:492] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P 14f33dcab292446f9ee74b2a2a56f56d: No bootstrap required, opened a new log
I20260619 08:56:34.781054 7833 ts_tablet_manager.cc:1397] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P 14f33dcab292446f9ee74b2a2a56f56d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:34.781054 7834 tablet_bootstrap.cc:654] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P c2237f88ed954837b849c5727176aac3: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:34.781235 7836 tablet_bootstrap.cc:492] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P b314603327ad4e21bf7d82da8a17eaa3: Bootstrap starting.
I20260619 08:56:34.781205 7833 raft_consensus.cc:348] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P 14f33dcab292446f9ee74b2a2a56f56d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f33dcab292446f9ee74b2a2a56f56d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42913 } } peers { permanent_uuid: "b314603327ad4e21bf7d82da8a17eaa3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37349 } } peers { permanent_uuid: "c2237f88ed954837b849c5727176aac3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39819 } }
I20260619 08:56:34.781276 7833 raft_consensus.cc:374] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P 14f33dcab292446f9ee74b2a2a56f56d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:34.781303 7833 raft_consensus.cc:729] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P 14f33dcab292446f9ee74b2a2a56f56d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14f33dcab292446f9ee74b2a2a56f56d, State: Initialized, Role: FOLLOWER
I20260619 08:56:34.781347 7833 consensus_queue.cc:260] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P 14f33dcab292446f9ee74b2a2a56f56d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f33dcab292446f9ee74b2a2a56f56d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42913 } } peers { permanent_uuid: "b314603327ad4e21bf7d82da8a17eaa3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37349 } } peers { permanent_uuid: "c2237f88ed954837b849c5727176aac3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39819 } }
I20260619 08:56:34.781497 7833 ts_tablet_manager.cc:1428] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P 14f33dcab292446f9ee74b2a2a56f56d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:34.781944 7836 tablet_bootstrap.cc:654] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P b314603327ad4e21bf7d82da8a17eaa3: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:34.782123 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:34.781650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.98.2:42623)
0619 08:56:34.782045 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:34.782083 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":73,"threads_started":1}
I20260619 08:56:34.782414 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:34.782332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.98.2:46197)
0619 08:56:34.782372 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:34.782392 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 08:56:34.782501 7834 tablet_bootstrap.cc:492] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P c2237f88ed954837b849c5727176aac3: No bootstrap required, opened a new log
I20260619 08:56:34.782563 7834 ts_tablet_manager.cc:1397] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P c2237f88ed954837b849c5727176aac3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:56:34.782548 7836 tablet_bootstrap.cc:492] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P b314603327ad4e21bf7d82da8a17eaa3: No bootstrap required, opened a new log
I20260619 08:56:34.782611 7836 ts_tablet_manager.cc:1397] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P b314603327ad4e21bf7d82da8a17eaa3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:34.782717 7834 raft_consensus.cc:348] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P c2237f88ed954837b849c5727176aac3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f33dcab292446f9ee74b2a2a56f56d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42913 } } peers { permanent_uuid: "b314603327ad4e21bf7d82da8a17eaa3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37349 } } peers { permanent_uuid: "c2237f88ed954837b849c5727176aac3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39819 } }
I20260619 08:56:34.782717 7836 raft_consensus.cc:348] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P b314603327ad4e21bf7d82da8a17eaa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f33dcab292446f9ee74b2a2a56f56d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42913 } } peers { permanent_uuid: "b314603327ad4e21bf7d82da8a17eaa3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37349 } } peers { permanent_uuid: "c2237f88ed954837b849c5727176aac3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39819 } }
I20260619 08:56:34.782783 7834 raft_consensus.cc:374] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P c2237f88ed954837b849c5727176aac3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:34.782804 7834 raft_consensus.cc:729] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P c2237f88ed954837b849c5727176aac3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2237f88ed954837b849c5727176aac3, State: Initialized, Role: FOLLOWER
I20260619 08:56:34.782806 7836 raft_consensus.cc:374] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P b314603327ad4e21bf7d82da8a17eaa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:34.782828 7836 raft_consensus.cc:729] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P b314603327ad4e21bf7d82da8a17eaa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b314603327ad4e21bf7d82da8a17eaa3, State: Initialized, Role: FOLLOWER
I20260619 08:56:34.782858 7834 consensus_queue.cc:260] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P c2237f88ed954837b849c5727176aac3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f33dcab292446f9ee74b2a2a56f56d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42913 } } peers { permanent_uuid: "b314603327ad4e21bf7d82da8a17eaa3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37349 } } peers { permanent_uuid: "c2237f88ed954837b849c5727176aac3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39819 } }
I20260619 08:56:34.782892 7836 consensus_queue.cc:260] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P b314603327ad4e21bf7d82da8a17eaa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f33dcab292446f9ee74b2a2a56f56d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42913 } } peers { permanent_uuid: "b314603327ad4e21bf7d82da8a17eaa3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37349 } } peers { permanent_uuid: "c2237f88ed954837b849c5727176aac3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39819 } }
I20260619 08:56:34.783010 7834 ts_tablet_manager.cc:1428] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P c2237f88ed954837b849c5727176aac3: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:34.783025 7836 ts_tablet_manager.cc:1428] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P b314603327ad4e21bf7d82da8a17eaa3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:34.783375 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:34.783165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.98.1:42779)
0619 08:56:34.783307 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:34.783344 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":78,"threads_started":1}
I20260619 08:56:34.783510 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:34.783269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.98.3:58907)
0619 08:56:34.783434 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:34.783469 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":70,"threads_started":1}
I20260619 08:56:34.783735 7735 heartbeater.cc:503] Master 127.0.98.61:46667 was elected leader, sending a full tablet report...
I20260619 08:56:34.783740 7462 heartbeater.cc:503] Master 127.0.98.61:46667 was elected leader, sending a full tablet report...
I20260619 08:56:34.783835 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:34.783672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.98.3:57067)
0619 08:56:34.783787 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:34.783810 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
I20260619 08:56:34.784032 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:34.783544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.98.1:53381)
0619 08:56:34.783968 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:34.784003 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":66,"threads_started":1}
I20260619 08:56:34.867377 7841 raft_consensus.cc:482] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P c2237f88ed954837b849c5727176aac3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:56:34.867477 7841 raft_consensus.cc:504] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P c2237f88ed954837b849c5727176aac3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f33dcab292446f9ee74b2a2a56f56d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42913 } } peers { permanent_uuid: "b314603327ad4e21bf7d82da8a17eaa3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37349 } } peers { permanent_uuid: "c2237f88ed954837b849c5727176aac3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39819 } }
I20260619 08:56:34.868150 7841 leader_election.cc:288] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P c2237f88ed954837b849c5727176aac3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14f33dcab292446f9ee74b2a2a56f56d (127.0.98.2:42913), b314603327ad4e21bf7d82da8a17eaa3 (127.0.98.1:37349)
I20260619 08:56:34.870510 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:34.868016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34755 (local address 127.0.98.1:37349)
0619 08:56:34.868102 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:34.868104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:34.868163 (+ 59us) server_negotiation.cc:407] Connection header received
0619 08:56:34.868236 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:34.868237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:34.868249 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:34.868294 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:34.868430 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:34.868895 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:34.869843 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:34.870209 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:34.870267 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:34.870278 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:34.870409 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 08:56:34.870445 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 08:56:34.870513 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:34.868017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50363 (local address 127.0.98.2:42913)
0619 08:56:34.868131 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:34.868133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:34.868141 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:56:34.868170 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:34.868171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:34.868179 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:34.868228 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:34.868448 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:34.868913 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:34.869852 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:34.870213 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:34.870271 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:34.870282 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:34.870413 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 08:56:34.870449 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260619 08:56:34.870522 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:34.867951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42913 (local address 127.0.98.3:50363)
0619 08:56:34.868014 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:34.868026 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:34.868076 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:34.868288 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:34.868289 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:34.868291 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:34.868349 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:34.868350 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:34.868989 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:34.868990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:34.869765 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:34.869767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:34.870287 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:34.870288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:34.870352 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:34.870355 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:56:34.870407 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 08:56:34.870445 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":44}
I20260619 08:56:34.870532 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:34.867951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37349 (local address 127.0.98.3:34755)
0619 08:56:34.868098 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:34.868109 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:34.868173 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:34.868314 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:34.868315 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:34.868317 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:34.868362 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:34.868364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:34.868974 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:34.868975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:34.869758 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:34.869760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:34.870280 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:34.870281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:34.870345 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:34.870350 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 08:56:34.870396 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 08:56:34.870435 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":20}
I20260619 08:56:34.870815 7551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4a08c7d3b7a4ce48b18f79ff41e6ef5" candidate_uuid: "c2237f88ed954837b849c5727176aac3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14f33dcab292446f9ee74b2a2a56f56d" is_pre_election: true
I20260619 08:56:34.870915 7551 raft_consensus.cc:2454] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P 14f33dcab292446f9ee74b2a2a56f56d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2237f88ed954837b849c5727176aac3 in term 0.
I20260619 08:56:34.871068 7623 leader_election.cc:302] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P c2237f88ed954837b849c5727176aac3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14f33dcab292446f9ee74b2a2a56f56d, c2237f88ed954837b849c5727176aac3; no voters:
I20260619 08:56:34.871070 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4a08c7d3b7a4ce48b18f79ff41e6ef5" candidate_uuid: "c2237f88ed954837b849c5727176aac3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b314603327ad4e21bf7d82da8a17eaa3" is_pre_election: true
I20260619 08:56:34.871188 7414 raft_consensus.cc:2454] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P b314603327ad4e21bf7d82da8a17eaa3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2237f88ed954837b849c5727176aac3 in term 0.
I20260619 08:56:34.871194 7841 raft_consensus.cc:2781] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P c2237f88ed954837b849c5727176aac3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:56:34.871228 7841 raft_consensus.cc:482] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P c2237f88ed954837b849c5727176aac3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:56:34.871245 7841 raft_consensus.cc:3037] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P c2237f88ed954837b849c5727176aac3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:34.871881 7841 raft_consensus.cc:504] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P c2237f88ed954837b849c5727176aac3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f33dcab292446f9ee74b2a2a56f56d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42913 } } peers { permanent_uuid: "b314603327ad4e21bf7d82da8a17eaa3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37349 } } peers { permanent_uuid: "c2237f88ed954837b849c5727176aac3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39819 } }
I20260619 08:56:34.872005 7841 leader_election.cc:288] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P c2237f88ed954837b849c5727176aac3 [CANDIDATE]: Term 1 election: Requested vote from peers 14f33dcab292446f9ee74b2a2a56f56d (127.0.98.2:42913), b314603327ad4e21bf7d82da8a17eaa3 (127.0.98.1:37349)
I20260619 08:56:34.872190 7551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4a08c7d3b7a4ce48b18f79ff41e6ef5" candidate_uuid: "c2237f88ed954837b849c5727176aac3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14f33dcab292446f9ee74b2a2a56f56d"
I20260619 08:56:34.872265 7551 raft_consensus.cc:3037] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P 14f33dcab292446f9ee74b2a2a56f56d [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:34.872841 7551 raft_consensus.cc:2454] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P 14f33dcab292446f9ee74b2a2a56f56d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2237f88ed954837b849c5727176aac3 in term 1.
I20260619 08:56:34.872938 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4a08c7d3b7a4ce48b18f79ff41e6ef5" candidate_uuid: "c2237f88ed954837b849c5727176aac3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b314603327ad4e21bf7d82da8a17eaa3"
I20260619 08:56:34.873003 7414 raft_consensus.cc:3037] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P b314603327ad4e21bf7d82da8a17eaa3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:34.873030 7623 leader_election.cc:302] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P c2237f88ed954837b849c5727176aac3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14f33dcab292446f9ee74b2a2a56f56d, c2237f88ed954837b849c5727176aac3; no voters:
I20260619 08:56:34.873135 7841 raft_consensus.cc:2781] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P c2237f88ed954837b849c5727176aac3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:34.873340 7841 raft_consensus.cc:686] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P c2237f88ed954837b849c5727176aac3 [term 1 LEADER]: Becoming Leader. State: Replica: c2237f88ed954837b849c5727176aac3, State: Running, Role: LEADER
I20260619 08:56:34.873416 7841 consensus_queue.cc:237] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P c2237f88ed954837b849c5727176aac3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f33dcab292446f9ee74b2a2a56f56d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42913 } } peers { permanent_uuid: "b314603327ad4e21bf7d82da8a17eaa3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37349 } } peers { permanent_uuid: "c2237f88ed954837b849c5727176aac3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39819 } }
I20260619 08:56:34.873725 7414 raft_consensus.cc:2454] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P b314603327ad4e21bf7d82da8a17eaa3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2237f88ed954837b849c5727176aac3 in term 1.
I20260619 08:56:34.873878 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:34.873645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.98.3:43023)
0619 08:56:34.873812 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:34.873848 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260619 08:56:34.874167 7094 catalog_manager.cc:5495] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P c2237f88ed954837b849c5727176aac3 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2237f88ed954837b849c5727176aac3 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c2237f88ed954837b849c5727176aac3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14f33dcab292446f9ee74b2a2a56f56d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b314603327ad4e21bf7d82da8a17eaa3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2237f88ed954837b849c5727176aac3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39819 } health_report { overall_health: HEALTHY } } }
I20260619 08:56:34.894789 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:34.892735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37924 (local address 127.0.98.3:39819)
0619 08:56:34.892777 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:34.892778 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:34.892785 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:56:34.892816 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:34.892816 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:34.892835 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:34.892892 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:56:34.892982 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:34.893451 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:34.894165 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:34.894352 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:34.894429 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:34.894534 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:56:34.894629 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:56:34.894677 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:34.894706 (+ 29us) server_negotiation.cc:299] Negotiation successful
0619 08:56:34.894722 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 08:56:34.895808 7551 raft_consensus.cc:1261] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P 14f33dcab292446f9ee74b2a2a56f56d [term 1 FOLLOWER]: Refusing update from remote peer c2237f88ed954837b849c5727176aac3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:56:34.895871 7414 raft_consensus.cc:1261] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P b314603327ad4e21bf7d82da8a17eaa3 [term 1 FOLLOWER]: Refusing update from remote peer c2237f88ed954837b849c5727176aac3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:56:34.896126 7841 consensus_queue.cc:1035] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P c2237f88ed954837b849c5727176aac3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14f33dcab292446f9ee74b2a2a56f56d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:56:34.896272 7847 consensus_queue.cc:1035] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P c2237f88ed954837b849c5727176aac3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b314603327ad4e21bf7d82da8a17eaa3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:56:34.896926 7599 heartbeater.cc:503] Master 127.0.98.61:46667 was elected leader, sending a full tablet report...
I20260619 08:56:34.897413 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:34.897315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.98.2:35529)
0619 08:56:34.897353 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:34.897382 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260619 08:56:34.897995 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:34.897908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.98.3:60221)
0619 08:56:34.897944 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:34.897968 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260619 08:56:34.898267 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:34.898164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.98.1:51125)
0619 08:56:34.898203 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:34.898236 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260619 08:56:34.898695 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:34.898609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.98.2:35419)
0619 08:56:34.898645 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:34.898667 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260619 08:56:34.898708 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:34.898632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.98.3:39005)
0619 08:56:34.898664 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:34.898686 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260619 08:56:34.899216 7852 mvcc.cc:205] Tried to move back new op lower bound from 7298496081491079168 to 7298496081401798656. Current Snapshot: MvccSnapshot[applied={T|T < 7298496081491079168}]
I20260619 08:56:34.900290 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:34.900209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39107 (local address 127.0.98.1:60913)
0619 08:56:34.900241 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:34.900263 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260619 08:56:34.900254 7094 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:56:34.900334 7094 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:56:34.901458 7094 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3ff9d07cc16e4b48aff2249b4cdfdf46: Sending DeleteTablet for 3 replicas of tablet b4a08c7d3b7a4ce48b18f79ff41e6ef5
I20260619 08:56:34.901799 7531 tablet_service.cc:1514] Processing DeleteTablet for tablet b4a08c7d3b7a4ce48b18f79ff41e6ef5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:34 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:56136
I20260619 08:56:34.901970 7859 tablet_replica.cc:330] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P 14f33dcab292446f9ee74b2a2a56f56d: stopping tablet replica
I20260619 08:56:34.902005 7667 tablet_service.cc:1514] Processing DeleteTablet for tablet b4a08c7d3b7a4ce48b18f79ff41e6ef5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:34 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:37908
I20260619 08:56:34.902022 7859 raft_consensus.cc:2229] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P 14f33dcab292446f9ee74b2a2a56f56d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:56:34.902024 7394 tablet_service.cc:1514] Processing DeleteTablet for tablet b4a08c7d3b7a4ce48b18f79ff41e6ef5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:34 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:43972
I20260619 08:56:34.902056 7859 raft_consensus.cc:2258] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P 14f33dcab292446f9ee74b2a2a56f56d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:34.902149 7860 tablet_replica.cc:330] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P c2237f88ed954837b849c5727176aac3: stopping tablet replica
I20260619 08:56:34.902190 7860 raft_consensus.cc:2229] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P c2237f88ed954837b849c5727176aac3 [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:56:34.902261 7860 raft_consensus.cc:2258] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P c2237f88ed954837b849c5727176aac3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:34.902395 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 7241
I20260619 08:56:34.904043 7859 ts_tablet_manager.cc:1905] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P 14f33dcab292446f9ee74b2a2a56f56d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:34.904768 7860 ts_tablet_manager.cc:1905] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P c2237f88ed954837b849c5727176aac3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:34.905642 7859 ts_tablet_manager.cc:1918] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P 14f33dcab292446f9ee74b2a2a56f56d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:56:34.905707 7859 log.cc:1196] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P 14f33dcab292446f9ee74b2a2a56f56d: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-1/wal/wals/b4a08c7d3b7a4ce48b18f79ff41e6ef5
I20260619 08:56:34.905902 7859 ts_tablet_manager.cc:1939] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P 14f33dcab292446f9ee74b2a2a56f56d: Deleting consensus metadata
I20260619 08:56:34.906150 7081 catalog_manager.cc:4841] TS 14f33dcab292446f9ee74b2a2a56f56d (127.0.98.2:42913): tablet b4a08c7d3b7a4ce48b18f79ff41e6ef5 (table test-table [id=f33dbd42cc964795b6fa178c1537781b]) successfully deleted
I20260619 08:56:34.906389 7860 ts_tablet_manager.cc:1918] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P c2237f88ed954837b849c5727176aac3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:56:34.906451 7860 log.cc:1196] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P c2237f88ed954837b849c5727176aac3: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859361540516-392-0/minicluster-data/ts-2/wal/wals/b4a08c7d3b7a4ce48b18f79ff41e6ef5
I20260619 08:56:34.906651 7860 ts_tablet_manager.cc:1939] T b4a08c7d3b7a4ce48b18f79ff41e6ef5 P c2237f88ed954837b849c5727176aac3: Deleting consensus metadata
I20260619 08:56:34.906929 7078 catalog_manager.cc:4841] TS c2237f88ed954837b849c5727176aac3 (127.0.98.3:39819): tablet b4a08c7d3b7a4ce48b18f79ff41e6ef5 (table test-table [id=f33dbd42cc964795b6fa178c1537781b]) successfully deleted
W20260619 08:56:34.909268 7830 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:34.909100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37349 (local address 127.0.0.1:43982)
0619 08:56:34.909192 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:34.909220 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37349: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 08:56:34.909371 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 7468
W20260619 08:56:34.909375 7078 catalog_manager.cc:4584] TS b314603327ad4e21bf7d82da8a17eaa3 (127.0.98.1:37349): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b4a08c7d3b7a4ce48b18f79ff41e6ef5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37349: connect: Connection reset by peer (error 104)
I20260619 08:56:34.914129 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 7604
I20260619 08:56:34.918471 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 6960
I20260619 08:56:34.922215 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 7148
2026-06-19T08:56:34Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6621 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:56:34 dist-test-slave-h4g5 krb5kdc[7864](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:56:34 dist-test-slave-h4g5 krb5kdc[7864](info): set up 2 sockets
Jun 19 08:56:34 dist-test-slave-h4g5 krb5kdc[7864](info): commencing operation
krb5kdc: starting...
W20260619 08:56:35.880256 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 08:56:35 dist-test-slave-h4g5 krb5kdc[7864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:56:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:56:35Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260619 08:56:35.992705 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44545
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44545
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:36.073360 7877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:36.073591 7877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:36.073622 7877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:36.074831 7877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:56:36.074877 7877 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 08:56:36.074900 7877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:36.074918 7877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:36.074932 7877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:56:36.074949 7877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:56:36.076836 7877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44545
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:36.077237 7877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:36.077490 7877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 08:56:36.082845 7877 server_base.cc:1034] running on GCE node
W20260619 08:56:36.082798 7882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:36.082809 7883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:36.082823 7885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:36.083323 7877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:36.083621 7877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:36.084770 7877 hybrid_clock.cc:648] HybridClock initialized: now 1781859396084755 us; error 32 us; skew 500 ppm
Jun 19 08:56:36 dist-test-slave-h4g5 krb5kdc[7864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859396, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:36.087251 7877 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260619 08:56:36.087728 7877 webserver.cc:460] Webserver started at http://127.0.98.62:35733/ using document root <none> and password file <none>
I20260619 08:56:36.087944 7877 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:36.087990 7877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:36.088153 7877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:36.089176 7877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "57ff80e31aa34b1d880dacffdefa69ab"
format_stamp: "Formatted at 2026-06-19 08:56:36 on dist-test-slave-h4g5"
server_key: "bed1fbf7a55bb3a0d079b64ecb015e4f"
server_key_iv: "2f6afcd3584b28859718d85e0117e7af"
server_key_version: "encryptionkey@0"
I20260619 08:56:36.089490 7877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "57ff80e31aa34b1d880dacffdefa69ab"
format_stamp: "Formatted at 2026-06-19 08:56:36 on dist-test-slave-h4g5"
server_key: "bed1fbf7a55bb3a0d079b64ecb015e4f"
server_key_iv: "2f6afcd3584b28859718d85e0117e7af"
server_key_version: "encryptionkey@0"
I20260619 08:56:36.090796 7877 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:56:36.091611 7892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:36.091828 7877 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:36.091919 7877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "57ff80e31aa34b1d880dacffdefa69ab"
format_stamp: "Formatted at 2026-06-19 08:56:36 on dist-test-slave-h4g5"
server_key: "bed1fbf7a55bb3a0d079b64ecb015e4f"
server_key_iv: "2f6afcd3584b28859718d85e0117e7af"
server_key_version: "encryptionkey@0"
I20260619 08:56:36.091991 7877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:36.131882 7877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:36.133963 7877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:36.134101 7877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:36.138403 7877 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44545
I20260619 08:56:36.138450 7954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44545 every 8 connection(s)
I20260619 08:56:36.138759 7877 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:56:36.139315 7955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:36.141656 7955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57ff80e31aa34b1d880dacffdefa69ab: Bootstrap starting.
I20260619 08:56:36.142205 7955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57ff80e31aa34b1d880dacffdefa69ab: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:36.142465 7955 log.cc:824] T 00000000000000000000000000000000 P 57ff80e31aa34b1d880dacffdefa69ab: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:36.143086 7955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57ff80e31aa34b1d880dacffdefa69ab: No bootstrap required, opened a new log
I20260619 08:56:36.145121 7955 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57ff80e31aa34b1d880dacffdefa69ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ff80e31aa34b1d880dacffdefa69ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } }
I20260619 08:56:36.145258 7955 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57ff80e31aa34b1d880dacffdefa69ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:36.145279 7955 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57ff80e31aa34b1d880dacffdefa69ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57ff80e31aa34b1d880dacffdefa69ab, State: Initialized, Role: FOLLOWER
I20260619 08:56:36.145406 7955 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57ff80e31aa34b1d880dacffdefa69ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ff80e31aa34b1d880dacffdefa69ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } }
I20260619 08:56:36.145474 7955 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57ff80e31aa34b1d880dacffdefa69ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:56:36.145495 7955 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57ff80e31aa34b1d880dacffdefa69ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:56:36.145531 7955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57ff80e31aa34b1d880dacffdefa69ab [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:36.146044 7955 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57ff80e31aa34b1d880dacffdefa69ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ff80e31aa34b1d880dacffdefa69ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } }
I20260619 08:56:36.146129 7955 leader_election.cc:302] T 00000000000000000000000000000000 P 57ff80e31aa34b1d880dacffdefa69ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57ff80e31aa34b1d880dacffdefa69ab; no voters:
I20260619 08:56:36.146277 7955 leader_election.cc:288] T 00000000000000000000000000000000 P 57ff80e31aa34b1d880dacffdefa69ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:56:36.146325 7958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57ff80e31aa34b1d880dacffdefa69ab [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:36.146507 7958 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57ff80e31aa34b1d880dacffdefa69ab [term 1 LEADER]: Becoming Leader. State: Replica: 57ff80e31aa34b1d880dacffdefa69ab, State: Running, Role: LEADER
I20260619 08:56:36.146524 7955 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57ff80e31aa34b1d880dacffdefa69ab [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:56:36.146643 7958 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57ff80e31aa34b1d880dacffdefa69ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ff80e31aa34b1d880dacffdefa69ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } }
I20260619 08:56:36.147022 7960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57ff80e31aa34b1d880dacffdefa69ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57ff80e31aa34b1d880dacffdefa69ab. Latest consensus state: current_term: 1 leader_uuid: "57ff80e31aa34b1d880dacffdefa69ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ff80e31aa34b1d880dacffdefa69ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } }
I20260619 08:56:36.147192 7959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57ff80e31aa34b1d880dacffdefa69ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57ff80e31aa34b1d880dacffdefa69ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57ff80e31aa34b1d880dacffdefa69ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44545 } } }
I20260619 08:56:36.147259 7960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57ff80e31aa34b1d880dacffdefa69ab [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:36.147315 7959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57ff80e31aa34b1d880dacffdefa69ab [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:36.147630 7968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:56:36.147887 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 7877
I20260619 08:56:36.147971 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 08:56:36.148103 7968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:56:36.148207 392 external_mini_cluster.cc:1428] Setting key 94fbd1dd8f71998afa539c64e12b7465
I20260619 08:56:36.150138 7968 catalog_manager.cc:1347] Generated new cluster ID: 469681799fc04e55982a0117220ae2f3
I20260619 08:56:36.150189 7968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 08:56:36 dist-test-slave-h4g5 krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 08:56:36.157733 7968 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:56:36.157789 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:36.148793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47422 (local address 127.0.98.62:44545)
0619 08:56:36.149380 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:36.149388 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:36.149430 (+ 42us) server_negotiation.cc:407] Connection header received
0619 08:56:36.149540 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:36.149546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:36.149598 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:36.149733 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:36.150340 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.151278 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.152029 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.152513 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.152618 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:36.154753 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:36.154757 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:36.154758 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:36.154770 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:36.156347 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:36.156662 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:36.156665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:36.156666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:36.156748 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:36.156916 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:36.156917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:36.156917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:36.157143 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:36.157286 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:36.157439 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 08:56:36.157558 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":60,"threads_started":1}
I20260619 08:56:36.158408 7968 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:56:36.161653 7968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57ff80e31aa34b1d880dacffdefa69ab: Generated new TSK 0
I20260619 08:56:36.161837 7968 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260619 08:56:36.190508 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44545
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:36.305750 7981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:36.306025 7981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:36.306063 7981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:36.307819 7981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:36.307881 7981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:36.307914 7981 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 08:56:36.307937 7981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 08:56:36.310209 7981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:36.310461 7981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:36.310709 7981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:36.315858 7987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:36.315843 7986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:36.315843 7989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:36.316004 7981 server_base.cc:1034] running on GCE node
I20260619 08:56:36.316380 7981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:36.316637 7981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:36.317803 7981 hybrid_clock.cc:648] HybridClock initialized: now 1781859396317781 us; error 41 us; skew 500 ppm
Jun 19 08:56:36 dist-test-slave-h4g5 krb5kdc[7864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859396, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:36.320286 7981 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 08:56:36.320693 7981 webserver.cc:460] Webserver started at http://127.0.98.1:41119/ using document root <none> and password file <none>
I20260619 08:56:36.320881 7981 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:36.320928 7981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:36.321019 7981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:36.322000 7981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-0/data/instance:
uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c"
format_stamp: "Formatted at 2026-06-19 08:56:36 on dist-test-slave-h4g5"
server_key: "97f2a8a68bd18d947539dc6d59cc9ca0"
server_key_iv: "ea650707bd87a1dd4ecdf8ffd4659d18"
server_key_version: "encryptionkey@0"
I20260619 08:56:36.322285 7981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-0/wal/instance:
uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c"
format_stamp: "Formatted at 2026-06-19 08:56:36 on dist-test-slave-h4g5"
server_key: "97f2a8a68bd18d947539dc6d59cc9ca0"
server_key_iv: "ea650707bd87a1dd4ecdf8ffd4659d18"
server_key_version: "encryptionkey@0"
I20260619 08:56:36.323472 7981 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:56:36.324156 7996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:36.324295 7981 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 08:56:36.324376 7981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-0/wal
uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c"
format_stamp: "Formatted at 2026-06-19 08:56:36 on dist-test-slave-h4g5"
server_key: "97f2a8a68bd18d947539dc6d59cc9ca0"
server_key_iv: "ea650707bd87a1dd4ecdf8ffd4659d18"
server_key_version: "encryptionkey@0"
I20260619 08:56:36.324431 7981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:36.333750 7981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:36.335723 7981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:36.335839 7981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:36.336383 7981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:36.336424 7981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:36.336445 7981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:36.336457 7981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:36.342335 7981 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45395
I20260619 08:56:36.342406 8111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45395 every 8 connection(s)
I20260619 08:56:36.342751 7981 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 08:56:36 dist-test-slave-h4g5 krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859396, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 08:56:36.346010 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:36.336792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40793 (local address 127.0.98.62:44545)
0619 08:56:36.336854 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:36.336856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:36.337065 (+ 209us) server_negotiation.cc:407] Connection header received
0619 08:56:36.337587 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:36.337589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:36.337606 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:36.337689 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:36.338512 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.339268 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.340747 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.341067 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.341159 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:36.343713 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:36.343714 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:36.343715 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:36.343724 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:36.344860 (+ 1136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:36.345262 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:36.345262 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:36.345263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:36.345307 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:36.345509 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:36.345509 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:36.345509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:36.345601 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:36.345646 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:36.345877 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 08:56:36.345940 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 08:56:36.346032 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 7981
I20260619 08:56:36.346141 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-0/wal/instance
I20260619 08:56:36.346275 8005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:36.336769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.98.1:40793)
0619 08:56:36.336938 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:36.336978 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:36.337468 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:36.337765 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:36.337769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:36.338132 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:36.338410 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:36.338416 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.339363 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.339364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:36.340661 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:36.340664 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.341183 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.341184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:36.341341 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:36.341954 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:36.341963 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:36.343632 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:36.344949 (+ 1317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:36.344956 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:36.344958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:36.345196 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:36.345355 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:36.345355 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:36.345355 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:36.345461 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:36.345649 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:36.345652 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:36.345767 (+ 115us) client_negotiation.cc:769] Sending connection context
0619 08:56:36.345877 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 08:56:36.346053 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":67,"threads_started":1}
I20260619 08:56:36.346340 392 external_mini_cluster.cc:1428] Setting key bdd8828ca1fba7be5f13f64773e6b68a
I20260619 08:56:36.346799 8112 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44545
I20260619 08:56:36.346885 8112 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:36.347097 8112 heartbeater.cc:511] Master 127.0.98.62:44545 requested a full tablet report, sending...
I20260619 08:56:36.347626 7909 ts_manager.cc:194] Registered new tserver with Master: 33cbeb4cbd1840b5a05436a1aef6bf2c (127.0.98.1:45395)
I20260619 08:56:36.348426 7909 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40793
I20260619 08:56:36.353549 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:36.348712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53581 (local address 127.0.98.62:44545)
0619 08:56:36.348768 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:36.348770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:36.348778 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:56:36.348797 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:36.348798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:36.348807 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:36.348857 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:36.349143 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.349611 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.350274 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.350465 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.350516 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:36.351399 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:36.351400 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:36.351401 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:36.351407 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:36.352606 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:36.352888 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:36.352889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:36.352890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:36.352936 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:36.353099 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:36.353100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:36.353100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:36.353201 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:36.353254 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:36.353383 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 08:56:36.353458 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 08:56:36.353569 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:36.347903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.98.1:53581)
0619 08:56:36.348032 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:36.348043 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:36.348126 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:36.348861 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:36.348863 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:36.349031 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:36.349080 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:36.349081 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.349695 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.349696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:36.350206 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:36.350207 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.350519 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.350520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:36.350579 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:36.351053 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:36.351057 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:36.351322 (+ 265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:36.352675 (+ 1353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:36.352676 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:36.352677 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:36.352827 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:36.352986 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:36.352987 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:36.352987 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:36.353051 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:36.353255 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:36.353256 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:36.353298 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 08:56:36.353357 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 08:56:36.353456 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":71,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260619 08:56:36.370253 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44545
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:36.448880 8121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:36.449081 8121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:36.449105 8121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:36.450244 8121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:36.450285 8121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:36.450305 8121 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 08:56:36.450323 8121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 08:56:36.451776 8121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:36.451994 8121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:36.452227 8121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:36.456871 8126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:36.456972 8121 server_base.cc:1034] running on GCE node
W20260619 08:56:36.456853 8129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:36.457062 8127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:36.457320 8121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:36.457535 8121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:36.458669 8121 hybrid_clock.cc:648] HybridClock initialized: now 1781859396458658 us; error 29 us; skew 500 ppm
Jun 19 08:56:36 dist-test-slave-h4g5 krb5kdc[7864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859396, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:36.460954 8121 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 08:56:36.461354 8121 webserver.cc:460] Webserver started at http://127.0.98.2:38989/ using document root <none> and password file <none>
I20260619 08:56:36.461555 8121 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:36.461603 8121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:36.461723 8121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:36.462733 8121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-1/data/instance:
uuid: "a770824c003e422f8fd8ef3d458332ee"
format_stamp: "Formatted at 2026-06-19 08:56:36 on dist-test-slave-h4g5"
server_key: "f01063f2b0b0cc78d4cbc6f9788185e2"
server_key_iv: "d87dbdef5cb66e9fcf3b0c830413476e"
server_key_version: "encryptionkey@0"
I20260619 08:56:36.463042 8121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a770824c003e422f8fd8ef3d458332ee"
format_stamp: "Formatted at 2026-06-19 08:56:36 on dist-test-slave-h4g5"
server_key: "f01063f2b0b0cc78d4cbc6f9788185e2"
server_key_iv: "d87dbdef5cb66e9fcf3b0c830413476e"
server_key_version: "encryptionkey@0"
I20260619 08:56:36.464323 8121 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:56:36.465083 8136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:36.465243 8121 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:36.465328 8121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-1/wal
uuid: "a770824c003e422f8fd8ef3d458332ee"
format_stamp: "Formatted at 2026-06-19 08:56:36 on dist-test-slave-h4g5"
server_key: "f01063f2b0b0cc78d4cbc6f9788185e2"
server_key_iv: "d87dbdef5cb66e9fcf3b0c830413476e"
server_key_version: "encryptionkey@0"
I20260619 08:56:36.465404 8121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:36.478073 8121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:36.479857 8121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:36.480003 8121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:36.480509 8121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:36.480553 8121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:36.480587 8121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:36.480618 8121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 08:56:36 dist-test-slave-h4g5 krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859396, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 08:56:36.487473 8121 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36709
I20260619 08:56:36.487553 8251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36709 every 8 connection(s)
I20260619 08:56:36.487887 8121 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
I20260619 08:56:36.490142 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:36.480937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36105 (local address 127.0.98.62:44545)
0619 08:56:36.481000 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:36.481002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:36.481153 (+ 151us) server_negotiation.cc:407] Connection header received
0619 08:56:36.481656 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:36.481657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:36.481670 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:36.481761 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:36.482454 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.483181 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.484712 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.484997 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.485079 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:36.487531 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:36.487532 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:36.487533 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:36.487541 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:36.488914 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:36.489268 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:36.489269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:36.489270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:36.489330 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:36.489537 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:36.489538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:36.489538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:36.489656 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:36.489712 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:36.489962 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 08:56:36.490041 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 08:56:36.490306 8145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:36.480893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.98.2:36105)
0619 08:56:36.481030 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:36.481068 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:36.481550 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:36.481816 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:36.481820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:36.482093 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:36.482360 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:36.482366 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.483303 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.483304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:36.484608 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:36.484611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.485100 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.485101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:36.485233 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:36.485746 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:36.485756 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:36.487422 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:36.488992 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:36.488998 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:36.489000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:36.489201 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:36.489381 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:36.489382 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:36.489382 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:36.489481 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:36.489708 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:36.489714 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:36.489846 (+ 132us) client_negotiation.cc:769] Sending connection context
0619 08:56:36.489932 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 08:56:36.490091 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":89,"threads_started":1}
I20260619 08:56:36.490732 8252 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44545
I20260619 08:56:36.490823 8252 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:36.491011 8252 heartbeater.cc:511] Master 127.0.98.62:44545 requested a full tablet report, sending...
I20260619 08:56:36.491400 7909 ts_manager.cc:194] Registered new tserver with Master: a770824c003e422f8fd8ef3d458332ee (127.0.98.2:36709)
I20260619 08:56:36.491866 7909 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36105
I20260619 08:56:36.494736 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 8121
I20260619 08:56:36.494853 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-1/wal/instance
I20260619 08:56:36.495090 392 external_mini_cluster.cc:1428] Setting key da3a49d89a9ae652fee1ecd352abafc8
I20260619 08:56:36.497522 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:36.491723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33517 (local address 127.0.98.62:44545)
0619 08:56:36.491760 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:36.491761 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:36.492140 (+ 379us) server_negotiation.cc:407] Connection header received
0619 08:56:36.492202 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:36.492203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:36.492214 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:36.492260 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:36.492618 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.493082 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.493738 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.493925 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.493977 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:36.495013 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:36.495014 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:36.495014 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:36.495020 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:36.496487 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:36.496770 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:36.496770 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:36.496771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:36.496825 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:36.496951 (+ 126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:36.496952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:36.496952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:36.497119 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:36.497184 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:36.497368 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 08:56:36.497439 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 08:56:36.497530 8258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:36.491670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.98.2:33517)
0619 08:56:36.492049 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:36.492060 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:36.492155 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:36.492277 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:36.492278 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:36.492479 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:36.492548 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:36.492550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.493165 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.493166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:36.493676 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:36.493678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.493980 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.493981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:36.494030 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:36.494497 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:36.494502 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:36.494874 (+ 372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:36.496560 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:36.496561 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:36.496561 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:36.496711 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:36.496875 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:36.496876 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:36.496876 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:36.496918 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:36.497215 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:36.497217 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:36.497263 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 08:56:36.497327 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 08:56:36.497424 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":64,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 08:56:36.516914 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44545
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:36.594622 8261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:36.594831 8261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:36.594863 8261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:36.596019 8261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:36.596060 8261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:36.596113 8261 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 08:56:36.596140 8261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 08:56:36.597589 8261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:36.597813 8261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:36.598011 8261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 08:56:36.602896 8261 server_base.cc:1034] running on GCE node
W20260619 08:56:36.602916 8266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:36.603034 8269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:36.603075 8267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:36.603269 8261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:36.603488 8261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:36.604630 8261 hybrid_clock.cc:648] HybridClock initialized: now 1781859396604634 us; error 44 us; skew 500 ppm
Jun 19 08:56:36 dist-test-slave-h4g5 krb5kdc[7864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859396, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:36.607012 8261 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 08:56:36.607447 8261 webserver.cc:460] Webserver started at http://127.0.98.3:43585/ using document root <none> and password file <none>
I20260619 08:56:36.607652 8261 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:36.607699 8261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:36.607816 8261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:36.608844 8261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-2/data/instance:
uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b"
format_stamp: "Formatted at 2026-06-19 08:56:36 on dist-test-slave-h4g5"
server_key: "f0b3b789fde9d1e7fef38a0290528df9"
server_key_iv: "ae0b77c1bcadb0a61d483166c809f093"
server_key_version: "encryptionkey@0"
I20260619 08:56:36.609157 8261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-2/wal/instance:
uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b"
format_stamp: "Formatted at 2026-06-19 08:56:36 on dist-test-slave-h4g5"
server_key: "f0b3b789fde9d1e7fef38a0290528df9"
server_key_iv: "ae0b77c1bcadb0a61d483166c809f093"
server_key_version: "encryptionkey@0"
I20260619 08:56:36.610360 8261 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 08:56:36.610993 8276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:36.611135 8261 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 08:56:36.611212 8261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-2/wal
uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b"
format_stamp: "Formatted at 2026-06-19 08:56:36 on dist-test-slave-h4g5"
server_key: "f0b3b789fde9d1e7fef38a0290528df9"
server_key_iv: "ae0b77c1bcadb0a61d483166c809f093"
server_key_version: "encryptionkey@0"
I20260619 08:56:36.611268 8261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:36.627673 8261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:36.629568 8261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:36.629700 8261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:36.630249 8261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:36.630290 8261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:36.630322 8261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:36.630364 8261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 08:56:36 dist-test-slave-h4g5 krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859396, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 08:56:36.636895 8261 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40723
I20260619 08:56:36.637225 8261 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
I20260619 08:56:36.637264 8391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40723 every 8 connection(s)
I20260619 08:56:36.639042 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:36.630602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46901 (local address 127.0.98.62:44545)
0619 08:56:36.630661 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:36.630664 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:36.630877 (+ 213us) server_negotiation.cc:407] Connection header received
0619 08:56:36.631376 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:36.631378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:36.631389 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:36.631439 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:36.632045 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.632593 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.634125 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.634311 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.634374 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:36.636445 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:36.636446 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:36.636446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:36.636454 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:36.637862 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:36.638269 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:36.638270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:36.638270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:36.638330 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:36.638541 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:36.638542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:36.638542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:36.638662 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:36.638718 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:36.638924 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 08:56:36.638980 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 08:56:36.639187 8285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:36.630592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.98.3:46901)
0619 08:56:36.630759 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:36.630798 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:36.631285 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:36.631505 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:36.631509 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:36.631783 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:36.631963 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:36.631967 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.632672 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.632673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:36.634042 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:36.634045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.634406 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.634407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:36.634543 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:36.635040 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:36.635050 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:36.636371 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:36.637925 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:36.637928 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:36.637929 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:36.638179 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:36.638387 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:36.638388 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:36.638388 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:36.638476 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:36.638713 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:36.638717 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:36.638824 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 08:56:36.638903 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 08:56:36.639015 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
I20260619 08:56:36.639614 8392 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44545
I20260619 08:56:36.639686 8392 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:36.639845 8392 heartbeater.cc:511] Master 127.0.98.62:44545 requested a full tablet report, sending...
I20260619 08:56:36.640269 7909 ts_manager.cc:194] Registered new tserver with Master: eac8f4fc9d344cee8cf11be42b0c0b3b (127.0.98.3:40723)
I20260619 08:56:36.640724 7909 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46901
I20260619 08:56:36.641054 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 8261
I20260619 08:56:36.641158 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-2/wal/instance
I20260619 08:56:36.641353 392 external_mini_cluster.cc:1428] Setting key da999da3d7c3fbcdd4d9a028ba78a7d3
I20260619 08:56:36.642063 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:56:36.645817 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:36.640573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32931 (local address 127.0.98.62:44545)
0619 08:56:36.640835 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:36.640837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:36.640842 (+ 5us) server_negotiation.cc:407] Connection header received
0619 08:56:36.640855 (+ 13us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:36.640855 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:36.640862 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:36.640904 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:36.641311 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.641838 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.642598 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.642838 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.642900 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:36.643673 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:36.643675 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:36.643675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:36.643682 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:36.644882 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:36.645169 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:36.645170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:36.645170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:36.645219 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:36.645373 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:36.645374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:36.645374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:36.645472 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:36.645523 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:36.645654 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 08:56:36.645719 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246}
I20260619 08:56:36.645824 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:36.640496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.98.3:32931)
0619 08:56:36.640614 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:36.640624 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:36.640671 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:36.640944 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:36.640945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:36.641170 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:36.641236 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:36.641238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.641909 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.641910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:36.642518 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:36.642520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.642906 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.642906 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:36.642950 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:36.643313 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:36.643317 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:36.643597 (+ 280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:36.644954 (+ 1357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:36.644955 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:36.644956 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:36.645111 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:36.645270 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:36.645271 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:36.645271 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:36.645325 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:36.645521 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:36.645522 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:36.645561 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 08:56:36.645627 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 08:56:36.645721 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":65,"threads_started":1}
Jun 19 08:56:36 dist-test-slave-h4g5 krb5kdc[7864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859396, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 08:56:36 dist-test-slave-h4g5 krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:36 dist-test-slave-h4g5 krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 08:56:36.661512 7977 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:36.656819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47436 (local address 127.0.98.62:44545)
0619 08:56:36.656860 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:36.656861 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:36.656870 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:56:36.656918 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:36.656918 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:36.656927 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:36.656972 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:36.657467 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.657969 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.658640 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.658806 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.658849 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:36.661432 (+ 2583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47436: BlockingRecv error: recv got EOF from 127.0.0.1:47436 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Jun 19 08:56:36 dist-test-slave-h4g5 krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859396, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 08:56:36.668437 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:36.662223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47446 (local address 127.0.98.62:44545)
0619 08:56:36.662260 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:36.662262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:36.662349 (+ 87us) server_negotiation.cc:407] Connection header received
0619 08:56:36.662410 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:36.662410 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:36.662419 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:36.662466 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:36.662939 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.663400 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.664024 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.664258 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.664306 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:36.666310 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:36.666311 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:36.666311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:36.666319 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:36.667369 (+ 1050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:36.667704 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:36.667705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:36.667706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:36.667759 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:36.667908 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:36.667909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:36.667909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:36.668051 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:36.668167 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:36.668285 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 08:56:36.668353 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 08:56:36.669776 7909 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:56:36.671278 7909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:56:36.678103 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:36.673486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44430 (local address 127.0.98.3:40723)
0619 08:56:36.673745 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:36.673750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:36.673760 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:56:36.673798 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:36.673800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:36.673816 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:36.673883 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:36.674112 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.675196 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.676796 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.677540 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.677593 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:36.677671 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:36.677952 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 08:56:36.677996 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":79,"threads_started":1}
I20260619 08:56:36.678198 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:36.673384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36709 (local address 127.0.0.1:33474)
0619 08:56:36.673735 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:36.673743 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:36.673782 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:36.673957 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:36.673958 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:36.673960 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:36.674019 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:36.674021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.675301 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.675302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:36.676946 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:36.676949 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.677945 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.677946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:36.678014 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:36.678018 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:56:36.678068 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 08:56:36.678131 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":69,"threads_started":1}
I20260619 08:56:36.678475 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:36.673380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40723 (local address 127.0.0.1:44430)
0619 08:56:36.673537 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:36.673576 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:36.673649 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:36.673952 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:36.673954 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:36.673957 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:36.674028 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:36.674030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.675554 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.675555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:36.676687 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:36.676690 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.677738 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.677739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:36.677820 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:36.677835 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 08:56:36.678354 (+ 519us) client_negotiation.cc:240] Negotiation successful
0619 08:56:36.678410 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":5,"thread_start_us":55,"threads_started":1}
I20260619 08:56:36.678753 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:36.673551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33474 (local address 127.0.98.2:36709)
0619 08:56:36.673711 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:36.673716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:36.673828 (+ 112us) server_negotiation.cc:407] Connection header received
0619 08:56:36.673867 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:36.673869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:36.673882 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:36.673948 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:36.674094 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.675200 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.677045 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.677760 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.677825 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:36.677912 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:36.678591 (+ 679us) server_negotiation.cc:299] Negotiation successful
0619 08:56:36.678649 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
I20260619 08:56:36.679396 8186 tablet_service.cc:1467] Processing CreateTablet for tablet 970eea57b7a043f4bd0c561e2d201b9a (DEFAULT_TABLE table=test-table [id=321f6e3a49cc4c1f93bc6ea9b4bfe03e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:36.679414 8309 tablet_service.cc:1467] Processing CreateTablet for tablet 970eea57b7a043f4bd0c561e2d201b9a (DEFAULT_TABLE table=test-table [id=321f6e3a49cc4c1f93bc6ea9b4bfe03e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:36.679677 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:36.673604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45395 (local address 127.0.0.1:60324)
0619 08:56:36.673793 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:36.673802 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:36.673854 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:36.675604 (+ 1750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:36.675605 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:36.675608 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:36.675677 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:36.675680 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.677058 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.677059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:36.678107 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:36.678110 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.679451 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.679452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:36.679511 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:36.679514 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:56:36.679561 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 08:56:36.679613 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":60,"threads_started":1}
I20260619 08:56:36.679736 8186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 970eea57b7a043f4bd0c561e2d201b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:36.679772 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 970eea57b7a043f4bd0c561e2d201b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:36.679966 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:36.673788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60324 (local address 127.0.98.1:45395)
0619 08:56:36.675353 (+ 1565us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:36.675357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:36.675365 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:56:36.675400 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:36.675403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:36.675419 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:36.675491 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:36.675773 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.676967 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.678205 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.679284 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.679348 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:36.679418 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:36.679841 (+ 423us) server_negotiation.cc:299] Negotiation successful
0619 08:56:36.679880 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1527,"thread_start_us":75,"threads_started":1}
I20260619 08:56:36.680541 8046 tablet_service.cc:1467] Processing CreateTablet for tablet 970eea57b7a043f4bd0c561e2d201b9a (DEFAULT_TABLE table=test-table [id=321f6e3a49cc4c1f93bc6ea9b4bfe03e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:36.680884 8046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 970eea57b7a043f4bd0c561e2d201b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:36.682544 8420 tablet_bootstrap.cc:492] T 970eea57b7a043f4bd0c561e2d201b9a P eac8f4fc9d344cee8cf11be42b0c0b3b: Bootstrap starting.
I20260619 08:56:36.683059 8420 tablet_bootstrap.cc:654] T 970eea57b7a043f4bd0c561e2d201b9a P eac8f4fc9d344cee8cf11be42b0c0b3b: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:36.683281 8420 log.cc:824] T 970eea57b7a043f4bd0c561e2d201b9a P eac8f4fc9d344cee8cf11be42b0c0b3b: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:36.683560 8421 tablet_bootstrap.cc:492] T 970eea57b7a043f4bd0c561e2d201b9a P a770824c003e422f8fd8ef3d458332ee: Bootstrap starting.
I20260619 08:56:36.683986 8421 tablet_bootstrap.cc:654] T 970eea57b7a043f4bd0c561e2d201b9a P a770824c003e422f8fd8ef3d458332ee: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:36.684306 8421 log.cc:824] T 970eea57b7a043f4bd0c561e2d201b9a P a770824c003e422f8fd8ef3d458332ee: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:36.684358 8423 tablet_bootstrap.cc:492] T 970eea57b7a043f4bd0c561e2d201b9a P 33cbeb4cbd1840b5a05436a1aef6bf2c: Bootstrap starting.
I20260619 08:56:36.684885 8423 tablet_bootstrap.cc:654] T 970eea57b7a043f4bd0c561e2d201b9a P 33cbeb4cbd1840b5a05436a1aef6bf2c: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:36.684990 8421 tablet_bootstrap.cc:492] T 970eea57b7a043f4bd0c561e2d201b9a P a770824c003e422f8fd8ef3d458332ee: No bootstrap required, opened a new log
I20260619 08:56:36.684990 8420 tablet_bootstrap.cc:492] T 970eea57b7a043f4bd0c561e2d201b9a P eac8f4fc9d344cee8cf11be42b0c0b3b: No bootstrap required, opened a new log
I20260619 08:56:36.685056 8420 ts_tablet_manager.cc:1397] T 970eea57b7a043f4bd0c561e2d201b9a P eac8f4fc9d344cee8cf11be42b0c0b3b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 08:56:36.685056 8421 ts_tablet_manager.cc:1397] T 970eea57b7a043f4bd0c561e2d201b9a P a770824c003e422f8fd8ef3d458332ee: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 08:56:36.685182 8423 log.cc:824] T 970eea57b7a043f4bd0c561e2d201b9a P 33cbeb4cbd1840b5a05436a1aef6bf2c: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:36.685737 8423 tablet_bootstrap.cc:492] T 970eea57b7a043f4bd0c561e2d201b9a P 33cbeb4cbd1840b5a05436a1aef6bf2c: No bootstrap required, opened a new log
I20260619 08:56:36.685791 8423 ts_tablet_manager.cc:1397] T 970eea57b7a043f4bd0c561e2d201b9a P 33cbeb4cbd1840b5a05436a1aef6bf2c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:56:36.686905 8420 raft_consensus.cc:348] T 970eea57b7a043f4bd0c561e2d201b9a P eac8f4fc9d344cee8cf11be42b0c0b3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } } peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } }
I20260619 08:56:36.686905 8421 raft_consensus.cc:348] T 970eea57b7a043f4bd0c561e2d201b9a P a770824c003e422f8fd8ef3d458332ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } } peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } }
I20260619 08:56:36.687063 8421 raft_consensus.cc:374] T 970eea57b7a043f4bd0c561e2d201b9a P a770824c003e422f8fd8ef3d458332ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:36.687103 8420 raft_consensus.cc:374] T 970eea57b7a043f4bd0c561e2d201b9a P eac8f4fc9d344cee8cf11be42b0c0b3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:36.687135 8421 raft_consensus.cc:729] T 970eea57b7a043f4bd0c561e2d201b9a P a770824c003e422f8fd8ef3d458332ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a770824c003e422f8fd8ef3d458332ee, State: Initialized, Role: FOLLOWER
I20260619 08:56:36.687160 8420 raft_consensus.cc:729] T 970eea57b7a043f4bd0c561e2d201b9a P eac8f4fc9d344cee8cf11be42b0c0b3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eac8f4fc9d344cee8cf11be42b0c0b3b, State: Initialized, Role: FOLLOWER
I20260619 08:56:36.687083 8423 raft_consensus.cc:348] T 970eea57b7a043f4bd0c561e2d201b9a P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } } peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } }
I20260619 08:56:36.687263 8423 raft_consensus.cc:374] T 970eea57b7a043f4bd0c561e2d201b9a P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:36.687281 8420 consensus_queue.cc:260] T 970eea57b7a043f4bd0c561e2d201b9a P eac8f4fc9d344cee8cf11be42b0c0b3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } } peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } }
I20260619 08:56:36.687270 8421 consensus_queue.cc:260] T 970eea57b7a043f4bd0c561e2d201b9a P a770824c003e422f8fd8ef3d458332ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } } peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } }
I20260619 08:56:36.687311 8423 raft_consensus.cc:729] T 970eea57b7a043f4bd0c561e2d201b9a P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33cbeb4cbd1840b5a05436a1aef6bf2c, State: Initialized, Role: FOLLOWER
I20260619 08:56:36.687434 8423 consensus_queue.cc:260] T 970eea57b7a043f4bd0c561e2d201b9a P 33cbeb4cbd1840b5a05436a1aef6bf2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } } peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } }
I20260619 08:56:36.687530 8421 ts_tablet_manager.cc:1428] T 970eea57b7a043f4bd0c561e2d201b9a P a770824c003e422f8fd8ef3d458332ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 08:56:36.687637 8252 heartbeater.cc:503] Master 127.0.98.62:44545 was elected leader, sending a full tablet report...
I20260619 08:56:36.687752 8423 ts_tablet_manager.cc:1428] T 970eea57b7a043f4bd0c561e2d201b9a P 33cbeb4cbd1840b5a05436a1aef6bf2c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:56:36.687795 8112 heartbeater.cc:503] Master 127.0.98.62:44545 was elected leader, sending a full tablet report...
I20260619 08:56:36.687530 8420 ts_tablet_manager.cc:1428] T 970eea57b7a043f4bd0c561e2d201b9a P eac8f4fc9d344cee8cf11be42b0c0b3b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 08:56:36.687988 8392 heartbeater.cc:503] Master 127.0.98.62:44545 was elected leader, sending a full tablet report...
W20260619 08:56:36.738526 8253 tablet.cc:2369] T 970eea57b7a043f4bd0c561e2d201b9a P a770824c003e422f8fd8ef3d458332ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:56:36.843262 8427 raft_consensus.cc:482] T 970eea57b7a043f4bd0c561e2d201b9a P a770824c003e422f8fd8ef3d458332ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:56:36.843405 8427 raft_consensus.cc:504] T 970eea57b7a043f4bd0c561e2d201b9a P a770824c003e422f8fd8ef3d458332ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } } peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } }
W20260619 08:56:36.843538 8113 tablet.cc:2369] T 970eea57b7a043f4bd0c561e2d201b9a P 33cbeb4cbd1840b5a05436a1aef6bf2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:56:36.843787 8427 leader_election.cc:288] T 970eea57b7a043f4bd0c561e2d201b9a P a770824c003e422f8fd8ef3d458332ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eac8f4fc9d344cee8cf11be42b0c0b3b (127.0.98.3:40723), 33cbeb4cbd1840b5a05436a1aef6bf2c (127.0.98.1:45395)
I20260619 08:56:36.846390 8145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:36.843962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40723 (local address 127.0.98.2:46955)
0619 08:56:36.844020 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:36.844031 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:36.844107 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:36.844251 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:36.844252 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:36.844254 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:36.844322 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:36.844323 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.844913 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.844914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:36.845657 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:36.845659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.846144 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.846145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:36.846231 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:36.846234 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:56:36.846277 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 08:56:36.846319 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 08:56:36.846390 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:36.844047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46955 (local address 127.0.98.3:40723)
0619 08:56:36.844136 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:36.844138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:36.844147 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:56:36.844175 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:36.844176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:36.844187 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:36.844234 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:36.844380 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.844844 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.845740 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.846077 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.846145 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:36.846155 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:36.846289 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 08:56:36.846324 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 08:56:36.846894 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:36.844101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45395 (local address 127.0.98.2:50837)
0619 08:56:36.844300 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:36.844308 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:36.844371 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:36.844512 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:36.844512 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:36.844514 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:36.844579 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:36.844581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.845159 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.845160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:36.845880 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:36.845882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.846636 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.846636 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:36.846707 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:36.846711 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:56:36.846799 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 08:56:36.846828 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":2,"thread_start_us":117,"threads_started":1}
I20260619 08:56:36.846909 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:36.844195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50837 (local address 127.0.98.1:45395)
0619 08:56:36.844252 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:36.844255 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:36.844357 (+ 102us) server_negotiation.cc:407] Connection header received
0619 08:56:36.844431 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:36.844432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:36.844443 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:36.844491 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:36.844634 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.845092 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.845962 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.846481 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.846582 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:36.846593 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:36.846808 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 08:56:36.846845 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 08:56:36.846881 8342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970eea57b7a043f4bd0c561e2d201b9a" candidate_uuid: "a770824c003e422f8fd8ef3d458332ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" is_pre_election: true
I20260619 08:56:36.847038 8342 raft_consensus.cc:2454] T 970eea57b7a043f4bd0c561e2d201b9a P eac8f4fc9d344cee8cf11be42b0c0b3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a770824c003e422f8fd8ef3d458332ee in term 0.
I20260619 08:56:36.847126 8050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970eea57b7a043f4bd0c561e2d201b9a" candidate_uuid: "a770824c003e422f8fd8ef3d458332ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" is_pre_election: true
I20260619 08:56:36.847259 8050 raft_consensus.cc:2454] T 970eea57b7a043f4bd0c561e2d201b9a P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a770824c003e422f8fd8ef3d458332ee in term 0.
I20260619 08:56:36.847267 8139 leader_election.cc:302] T 970eea57b7a043f4bd0c561e2d201b9a P a770824c003e422f8fd8ef3d458332ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a770824c003e422f8fd8ef3d458332ee, eac8f4fc9d344cee8cf11be42b0c0b3b; no voters:
I20260619 08:56:36.847388 8427 raft_consensus.cc:2781] T 970eea57b7a043f4bd0c561e2d201b9a P a770824c003e422f8fd8ef3d458332ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:56:36.847451 8427 raft_consensus.cc:482] T 970eea57b7a043f4bd0c561e2d201b9a P a770824c003e422f8fd8ef3d458332ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:56:36.847482 8427 raft_consensus.cc:3037] T 970eea57b7a043f4bd0c561e2d201b9a P a770824c003e422f8fd8ef3d458332ee [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:36.848497 8427 raft_consensus.cc:504] T 970eea57b7a043f4bd0c561e2d201b9a P a770824c003e422f8fd8ef3d458332ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } } peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } }
I20260619 08:56:36.848604 8427 leader_election.cc:288] T 970eea57b7a043f4bd0c561e2d201b9a P a770824c003e422f8fd8ef3d458332ee [CANDIDATE]: Term 1 election: Requested vote from peers eac8f4fc9d344cee8cf11be42b0c0b3b (127.0.98.3:40723), 33cbeb4cbd1840b5a05436a1aef6bf2c (127.0.98.1:45395)
I20260619 08:56:36.848762 8342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970eea57b7a043f4bd0c561e2d201b9a" candidate_uuid: "a770824c003e422f8fd8ef3d458332ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b"
I20260619 08:56:36.848783 8050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "970eea57b7a043f4bd0c561e2d201b9a" candidate_uuid: "a770824c003e422f8fd8ef3d458332ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c"
I20260619 08:56:36.848834 8342 raft_consensus.cc:3037] T 970eea57b7a043f4bd0c561e2d201b9a P eac8f4fc9d344cee8cf11be42b0c0b3b [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:36.848842 8050 raft_consensus.cc:3037] T 970eea57b7a043f4bd0c561e2d201b9a P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:36.849447 8050 raft_consensus.cc:2454] T 970eea57b7a043f4bd0c561e2d201b9a P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a770824c003e422f8fd8ef3d458332ee in term 1.
I20260619 08:56:36.849560 8342 raft_consensus.cc:2454] T 970eea57b7a043f4bd0c561e2d201b9a P eac8f4fc9d344cee8cf11be42b0c0b3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a770824c003e422f8fd8ef3d458332ee in term 1.
I20260619 08:56:36.849591 8139 leader_election.cc:302] T 970eea57b7a043f4bd0c561e2d201b9a P a770824c003e422f8fd8ef3d458332ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33cbeb4cbd1840b5a05436a1aef6bf2c, a770824c003e422f8fd8ef3d458332ee; no voters:
I20260619 08:56:36.849658 8427 raft_consensus.cc:2781] T 970eea57b7a043f4bd0c561e2d201b9a P a770824c003e422f8fd8ef3d458332ee [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:36.849778 8427 raft_consensus.cc:686] T 970eea57b7a043f4bd0c561e2d201b9a P a770824c003e422f8fd8ef3d458332ee [term 1 LEADER]: Becoming Leader. State: Replica: a770824c003e422f8fd8ef3d458332ee, State: Running, Role: LEADER
I20260619 08:56:36.849859 8427 consensus_queue.cc:237] T 970eea57b7a043f4bd0c561e2d201b9a P a770824c003e422f8fd8ef3d458332ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } } peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } }
I20260619 08:56:36.850669 7909 catalog_manager.cc:5495] T 970eea57b7a043f4bd0c561e2d201b9a P a770824c003e422f8fd8ef3d458332ee reported cstate change: term changed from 0 to 1, leader changed from <none> to a770824c003e422f8fd8ef3d458332ee (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a770824c003e422f8fd8ef3d458332ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } health_report { overall_health: UNKNOWN } } }
I20260619 08:56:36.859592 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:36.857343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33482 (local address 127.0.98.2:36709)
0619 08:56:36.857407 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:36.857409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:36.857417 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:56:36.857446 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:36.857447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:36.857464 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:36.857528 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:56:36.857688 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.858150 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.858900 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:36.859107 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:36.859175 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:36.859284 (+ 109us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:56:36.859380 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:56:36.859486 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:36.859500 (+ 14us) server_negotiation.cc:299] Negotiation successful
0619 08:56:36.859521 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 08:56:36.861346 8050 raft_consensus.cc:1261] T 970eea57b7a043f4bd0c561e2d201b9a P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 1 FOLLOWER]: Refusing update from remote peer a770824c003e422f8fd8ef3d458332ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:56:36.861368 8342 raft_consensus.cc:1261] T 970eea57b7a043f4bd0c561e2d201b9a P eac8f4fc9d344cee8cf11be42b0c0b3b [term 1 FOLLOWER]: Refusing update from remote peer a770824c003e422f8fd8ef3d458332ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:56:36.861652 8427 consensus_queue.cc:1035] T 970eea57b7a043f4bd0c561e2d201b9a P a770824c003e422f8fd8ef3d458332ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:56:36.861788 8430 consensus_queue.cc:1035] T 970eea57b7a043f4bd0c561e2d201b9a P a770824c003e422f8fd8ef3d458332ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:56:36.864584 8435 mvcc.cc:205] Tried to move back new op lower bound from 7298496089541304320 to 7298496089497337856. Current Snapshot: MvccSnapshot[applied={T|T < 7298496089541304320}]
I20260619 08:56:36.865857 7909 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:56:36.865947 7909 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:56:36.866575 7909 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57ff80e31aa34b1d880dacffdefa69ab: Sending DeleteTablet for 3 replicas of tablet 970eea57b7a043f4bd0c561e2d201b9a
I20260619 08:56:36.866876 8186 tablet_service.cc:1514] Processing DeleteTablet for tablet 970eea57b7a043f4bd0c561e2d201b9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33474
I20260619 08:56:36.866919 8309 tablet_service.cc:1514] Processing DeleteTablet for tablet 970eea57b7a043f4bd0c561e2d201b9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44430
I20260619 08:56:36.867084 8046 tablet_service.cc:1514] Processing DeleteTablet for tablet 970eea57b7a043f4bd0c561e2d201b9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60324
I20260619 08:56:36.867093 8443 tablet_replica.cc:330] T 970eea57b7a043f4bd0c561e2d201b9a P a770824c003e422f8fd8ef3d458332ee: stopping tablet replica
I20260619 08:56:36.867192 8443 raft_consensus.cc:2229] T 970eea57b7a043f4bd0c561e2d201b9a P a770824c003e422f8fd8ef3d458332ee [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:56:36.867179 7909 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 08:56:36.867280 8443 raft_consensus.cc:2258] T 970eea57b7a043f4bd0c561e2d201b9a P a770824c003e422f8fd8ef3d458332ee [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 08:56:36.867312 7909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:56:36.867440 8444 tablet_replica.cc:330] T 970eea57b7a043f4bd0c561e2d201b9a P eac8f4fc9d344cee8cf11be42b0c0b3b: stopping tablet replica
I20260619 08:56:36.867554 8445 tablet_replica.cc:330] T 970eea57b7a043f4bd0c561e2d201b9a P 33cbeb4cbd1840b5a05436a1aef6bf2c: stopping tablet replica
I20260619 08:56:36.867625 8444 raft_consensus.cc:2229] T 970eea57b7a043f4bd0c561e2d201b9a P eac8f4fc9d344cee8cf11be42b0c0b3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:56:36.868026 8444 raft_consensus.cc:2258] T 970eea57b7a043f4bd0c561e2d201b9a P eac8f4fc9d344cee8cf11be42b0c0b3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:36.868062 8443 ts_tablet_manager.cc:1905] T 970eea57b7a043f4bd0c561e2d201b9a P a770824c003e422f8fd8ef3d458332ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:36.868573 8444 ts_tablet_manager.cc:1905] T 970eea57b7a043f4bd0c561e2d201b9a P eac8f4fc9d344cee8cf11be42b0c0b3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:36.868973 8186 tablet_service.cc:1467] Processing CreateTablet for tablet 79004176209441dbba47b63cfe32927b (DEFAULT_TABLE table=test-table [id=186eced15a124c6db180fc2d742fe8cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:36.869109 8186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79004176209441dbba47b63cfe32927b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:36.869817 8443 ts_tablet_manager.cc:1918] T 970eea57b7a043f4bd0c561e2d201b9a P a770824c003e422f8fd8ef3d458332ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:56:36.869885 8443 log.cc:1196] T 970eea57b7a043f4bd0c561e2d201b9a P a770824c003e422f8fd8ef3d458332ee: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-1/wal/wals/970eea57b7a043f4bd0c561e2d201b9a
I20260619 08:56:36.870117 8443 ts_tablet_manager.cc:1939] T 970eea57b7a043f4bd0c561e2d201b9a P a770824c003e422f8fd8ef3d458332ee: Deleting consensus metadata
I20260619 08:56:36.870368 8421 tablet_bootstrap.cc:492] T 79004176209441dbba47b63cfe32927b P a770824c003e422f8fd8ef3d458332ee: Bootstrap starting.
I20260619 08:56:36.870421 8046 tablet_service.cc:1467] Processing CreateTablet for tablet 79004176209441dbba47b63cfe32927b (DEFAULT_TABLE table=test-table [id=186eced15a124c6db180fc2d742fe8cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:36.870527 8046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79004176209441dbba47b63cfe32927b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:36.870677 8444 ts_tablet_manager.cc:1918] T 970eea57b7a043f4bd0c561e2d201b9a P eac8f4fc9d344cee8cf11be42b0c0b3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:56:36.870762 8421 tablet_bootstrap.cc:654] T 79004176209441dbba47b63cfe32927b P a770824c003e422f8fd8ef3d458332ee: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:36.870780 8444 log.cc:1196] T 970eea57b7a043f4bd0c561e2d201b9a P eac8f4fc9d344cee8cf11be42b0c0b3b: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-2/wal/wals/970eea57b7a043f4bd0c561e2d201b9a
I20260619 08:56:36.871107 8444 ts_tablet_manager.cc:1939] T 970eea57b7a043f4bd0c561e2d201b9a P eac8f4fc9d344cee8cf11be42b0c0b3b: Deleting consensus metadata
I20260619 08:56:36.871532 7895 catalog_manager.cc:4841] TS eac8f4fc9d344cee8cf11be42b0c0b3b (127.0.98.3:40723): tablet 970eea57b7a043f4bd0c561e2d201b9a (table test-table [id=321f6e3a49cc4c1f93bc6ea9b4bfe03e]) successfully deleted
I20260619 08:56:36.871620 8423 tablet_bootstrap.cc:492] T 79004176209441dbba47b63cfe32927b P 33cbeb4cbd1840b5a05436a1aef6bf2c: Bootstrap starting.
I20260619 08:56:36.871660 8309 tablet_service.cc:1467] Processing CreateTablet for tablet 79004176209441dbba47b63cfe32927b (DEFAULT_TABLE table=test-table [id=186eced15a124c6db180fc2d742fe8cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:36.871750 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79004176209441dbba47b63cfe32927b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:36.871953 8423 tablet_bootstrap.cc:654] T 79004176209441dbba47b63cfe32927b P 33cbeb4cbd1840b5a05436a1aef6bf2c: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:36.872341 8421 tablet_bootstrap.cc:492] T 79004176209441dbba47b63cfe32927b P a770824c003e422f8fd8ef3d458332ee: No bootstrap required, opened a new log
I20260619 08:56:36.872416 8421 ts_tablet_manager.cc:1397] T 79004176209441dbba47b63cfe32927b P a770824c003e422f8fd8ef3d458332ee: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 08:56:36.872462 7893 catalog_manager.cc:4841] TS a770824c003e422f8fd8ef3d458332ee (127.0.98.2:36709): tablet 970eea57b7a043f4bd0c561e2d201b9a (table test-table [id=321f6e3a49cc4c1f93bc6ea9b4bfe03e]) successfully deleted
I20260619 08:56:36.872553 8423 tablet_bootstrap.cc:492] T 79004176209441dbba47b63cfe32927b P 33cbeb4cbd1840b5a05436a1aef6bf2c: No bootstrap required, opened a new log
I20260619 08:56:36.872591 8423 ts_tablet_manager.cc:1397] T 79004176209441dbba47b63cfe32927b P 33cbeb4cbd1840b5a05436a1aef6bf2c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 08:56:36.872574 8421 raft_consensus.cc:348] T 79004176209441dbba47b63cfe32927b P a770824c003e422f8fd8ef3d458332ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } }
I20260619 08:56:36.872651 8421 raft_consensus.cc:374] T 79004176209441dbba47b63cfe32927b P a770824c003e422f8fd8ef3d458332ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:36.872670 8421 raft_consensus.cc:729] T 79004176209441dbba47b63cfe32927b P a770824c003e422f8fd8ef3d458332ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a770824c003e422f8fd8ef3d458332ee, State: Initialized, Role: FOLLOWER
I20260619 08:56:36.872710 8421 consensus_queue.cc:260] T 79004176209441dbba47b63cfe32927b P a770824c003e422f8fd8ef3d458332ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } }
I20260619 08:56:36.872740 8423 raft_consensus.cc:348] T 79004176209441dbba47b63cfe32927b P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } }
I20260619 08:56:36.872787 8421 ts_tablet_manager.cc:1428] T 79004176209441dbba47b63cfe32927b P a770824c003e422f8fd8ef3d458332ee: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:36.872802 8423 raft_consensus.cc:374] T 79004176209441dbba47b63cfe32927b P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:36.872821 8423 raft_consensus.cc:729] T 79004176209441dbba47b63cfe32927b P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33cbeb4cbd1840b5a05436a1aef6bf2c, State: Initialized, Role: FOLLOWER
I20260619 08:56:36.872859 8423 consensus_queue.cc:260] T 79004176209441dbba47b63cfe32927b P 33cbeb4cbd1840b5a05436a1aef6bf2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } }
I20260619 08:56:36.872954 8423 ts_tablet_manager.cc:1428] T 79004176209441dbba47b63cfe32927b P 33cbeb4cbd1840b5a05436a1aef6bf2c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:36.873574 8420 tablet_bootstrap.cc:492] T 79004176209441dbba47b63cfe32927b P eac8f4fc9d344cee8cf11be42b0c0b3b: Bootstrap starting.
I20260619 08:56:36.873942 8420 tablet_bootstrap.cc:654] T 79004176209441dbba47b63cfe32927b P eac8f4fc9d344cee8cf11be42b0c0b3b: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:36.874183 8445 raft_consensus.cc:2229] T 970eea57b7a043f4bd0c561e2d201b9a P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:56:36.874302 8445 raft_consensus.cc:2258] T 970eea57b7a043f4bd0c561e2d201b9a P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:36.874599 8420 tablet_bootstrap.cc:492] T 79004176209441dbba47b63cfe32927b P eac8f4fc9d344cee8cf11be42b0c0b3b: No bootstrap required, opened a new log
I20260619 08:56:36.874645 8420 ts_tablet_manager.cc:1397] T 79004176209441dbba47b63cfe32927b P eac8f4fc9d344cee8cf11be42b0c0b3b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:36.874790 8445 ts_tablet_manager.cc:1905] T 970eea57b7a043f4bd0c561e2d201b9a P 33cbeb4cbd1840b5a05436a1aef6bf2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:36.874782 8420 raft_consensus.cc:348] T 79004176209441dbba47b63cfe32927b P eac8f4fc9d344cee8cf11be42b0c0b3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } }
I20260619 08:56:36.874840 8420 raft_consensus.cc:374] T 79004176209441dbba47b63cfe32927b P eac8f4fc9d344cee8cf11be42b0c0b3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:36.874863 8420 raft_consensus.cc:729] T 79004176209441dbba47b63cfe32927b P eac8f4fc9d344cee8cf11be42b0c0b3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eac8f4fc9d344cee8cf11be42b0c0b3b, State: Initialized, Role: FOLLOWER
I20260619 08:56:36.874898 8420 consensus_queue.cc:260] T 79004176209441dbba47b63cfe32927b P eac8f4fc9d344cee8cf11be42b0c0b3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } }
I20260619 08:56:36.874979 8420 ts_tablet_manager.cc:1428] T 79004176209441dbba47b63cfe32927b P eac8f4fc9d344cee8cf11be42b0c0b3b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:36.876139 8445 ts_tablet_manager.cc:1918] T 970eea57b7a043f4bd0c561e2d201b9a P 33cbeb4cbd1840b5a05436a1aef6bf2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:56:36.876196 8445 log.cc:1196] T 970eea57b7a043f4bd0c561e2d201b9a P 33cbeb4cbd1840b5a05436a1aef6bf2c: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-0/wal/wals/970eea57b7a043f4bd0c561e2d201b9a
I20260619 08:56:36.876623 8445 ts_tablet_manager.cc:1939] T 970eea57b7a043f4bd0c561e2d201b9a P 33cbeb4cbd1840b5a05436a1aef6bf2c: Deleting consensus metadata
I20260619 08:56:36.877003 7895 catalog_manager.cc:4841] TS 33cbeb4cbd1840b5a05436a1aef6bf2c (127.0.98.1:45395): tablet 970eea57b7a043f4bd0c561e2d201b9a (table test-table [id=321f6e3a49cc4c1f93bc6ea9b4bfe03e]) successfully deleted
W20260619 08:56:36.887734 8393 tablet.cc:2369] T 79004176209441dbba47b63cfe32927b P eac8f4fc9d344cee8cf11be42b0c0b3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:56:37.234819 8428 raft_consensus.cc:482] T 79004176209441dbba47b63cfe32927b P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:56:37.234932 8428 raft_consensus.cc:504] T 79004176209441dbba47b63cfe32927b P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } }
I20260619 08:56:37.235258 8428 leader_election.cc:288] T 79004176209441dbba47b63cfe32927b P 33cbeb4cbd1840b5a05436a1aef6bf2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a770824c003e422f8fd8ef3d458332ee (127.0.98.2:36709), eac8f4fc9d344cee8cf11be42b0c0b3b (127.0.98.3:40723)
I20260619 08:56:37.239329 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:37.235567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42053 (local address 127.0.98.2:36709)
0619 08:56:37.235607 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:37.235608 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:37.235693 (+ 85us) server_negotiation.cc:407] Connection header received
0619 08:56:37.235745 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:37.235745 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:37.235755 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:37.235811 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:37.236118 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:37.236860 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:37.238364 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:37.238912 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:37.238982 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:37.238992 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:37.239230 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 08:56:37.239266 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 08:56:37.239329 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:37.235529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54139 (local address 127.0.98.3:40723)
0619 08:56:37.235593 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:37.235595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:37.235649 (+ 54us) server_negotiation.cc:407] Connection header received
0619 08:56:37.235744 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:37.235745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:37.235755 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:37.235811 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:37.236108 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:37.236876 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:37.238363 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:37.238912 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:37.238982 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:37.238992 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:37.239193 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 08:56:37.239254 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 08:56:37.239383 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:37.235463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40723 (local address 127.0.98.1:54139)
0619 08:56:37.235587 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:37.235597 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:37.235662 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:37.235867 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:37.235868 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:37.235869 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:37.236014 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:37.236016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:37.236990 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:37.236990 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:37.238275 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:37.238278 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:37.239003 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:37.239004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:37.239117 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:37.239123 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 08:56:37.239180 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 08:56:37.239233 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":84,"thread_start_us":63,"threads_started":1}
I20260619 08:56:37.239388 8449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:37.235463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36709 (local address 127.0.98.1:42053)
0619 08:56:37.235587 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:37.235597 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:37.235689 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:37.235862 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:37.235863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:37.235867 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:37.236004 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:37.236007 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:37.236971 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:37.236972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:37.238275 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:37.238278 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:37.239027 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:37.239028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:37.239117 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:37.239125 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 08:56:37.239180 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 08:56:37.239233 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":44,"thread_start_us":59,"threads_started":1}
I20260619 08:56:37.239614 8342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79004176209441dbba47b63cfe32927b" candidate_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" is_pre_election: true
I20260619 08:56:37.239703 8342 raft_consensus.cc:2454] T 79004176209441dbba47b63cfe32927b P eac8f4fc9d344cee8cf11be42b0c0b3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33cbeb4cbd1840b5a05436a1aef6bf2c in term 0.
I20260619 08:56:37.239893 7999 leader_election.cc:302] T 79004176209441dbba47b63cfe32927b P 33cbeb4cbd1840b5a05436a1aef6bf2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33cbeb4cbd1840b5a05436a1aef6bf2c, eac8f4fc9d344cee8cf11be42b0c0b3b; no voters:
I20260619 08:56:37.239902 8206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79004176209441dbba47b63cfe32927b" candidate_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a770824c003e422f8fd8ef3d458332ee" is_pre_election: true
I20260619 08:56:37.240020 8206 raft_consensus.cc:2454] T 79004176209441dbba47b63cfe32927b P a770824c003e422f8fd8ef3d458332ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33cbeb4cbd1840b5a05436a1aef6bf2c in term 0.
I20260619 08:56:37.240051 8428 raft_consensus.cc:2781] T 79004176209441dbba47b63cfe32927b P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:56:37.240115 8428 raft_consensus.cc:482] T 79004176209441dbba47b63cfe32927b P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:56:37.240145 8428 raft_consensus.cc:3037] T 79004176209441dbba47b63cfe32927b P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:37.240833 8428 raft_consensus.cc:504] T 79004176209441dbba47b63cfe32927b P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } }
I20260619 08:56:37.240945 8428 leader_election.cc:288] T 79004176209441dbba47b63cfe32927b P 33cbeb4cbd1840b5a05436a1aef6bf2c [CANDIDATE]: Term 1 election: Requested vote from peers a770824c003e422f8fd8ef3d458332ee (127.0.98.2:36709), eac8f4fc9d344cee8cf11be42b0c0b3b (127.0.98.3:40723)
I20260619 08:56:37.241107 8206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79004176209441dbba47b63cfe32927b" candidate_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a770824c003e422f8fd8ef3d458332ee"
I20260619 08:56:37.241132 8342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79004176209441dbba47b63cfe32927b" candidate_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b"
I20260619 08:56:37.241182 8206 raft_consensus.cc:3037] T 79004176209441dbba47b63cfe32927b P a770824c003e422f8fd8ef3d458332ee [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:37.241199 8342 raft_consensus.cc:3037] T 79004176209441dbba47b63cfe32927b P eac8f4fc9d344cee8cf11be42b0c0b3b [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:37.241860 8342 raft_consensus.cc:2454] T 79004176209441dbba47b63cfe32927b P eac8f4fc9d344cee8cf11be42b0c0b3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33cbeb4cbd1840b5a05436a1aef6bf2c in term 1.
I20260619 08:56:37.241930 8206 raft_consensus.cc:2454] T 79004176209441dbba47b63cfe32927b P a770824c003e422f8fd8ef3d458332ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33cbeb4cbd1840b5a05436a1aef6bf2c in term 1.
I20260619 08:56:37.242038 7999 leader_election.cc:302] T 79004176209441dbba47b63cfe32927b P 33cbeb4cbd1840b5a05436a1aef6bf2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33cbeb4cbd1840b5a05436a1aef6bf2c, eac8f4fc9d344cee8cf11be42b0c0b3b; no voters:
I20260619 08:56:37.242144 8428 raft_consensus.cc:2781] T 79004176209441dbba47b63cfe32927b P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:37.242291 8428 raft_consensus.cc:686] T 79004176209441dbba47b63cfe32927b P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 1 LEADER]: Becoming Leader. State: Replica: 33cbeb4cbd1840b5a05436a1aef6bf2c, State: Running, Role: LEADER
I20260619 08:56:37.242364 8428 consensus_queue.cc:237] T 79004176209441dbba47b63cfe32927b P 33cbeb4cbd1840b5a05436a1aef6bf2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } }
I20260619 08:56:37.243026 7909 catalog_manager.cc:5495] T 79004176209441dbba47b63cfe32927b P 33cbeb4cbd1840b5a05436a1aef6bf2c reported cstate change: term changed from 0 to 1, leader changed from <none> to 33cbeb4cbd1840b5a05436a1aef6bf2c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } health_report { overall_health: UNKNOWN } } }
Jun 19 08:56:37 dist-test-slave-h4g5 krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859396, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 08:56:37.331100 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:37.324780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47448 (local address 127.0.98.62:44545)
0619 08:56:37.324911 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:37.324912 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:37.324945 (+ 33us) server_negotiation.cc:407] Connection header received
0619 08:56:37.325003 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:37.325004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:37.325013 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:37.325074 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:37.325494 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:37.326004 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:37.326669 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:37.326858 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:37.326911 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:37.328876 (+ 1965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:37.328877 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:37.328877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:37.328883 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:37.330027 (+ 1144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:37.330375 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:37.330376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:37.330376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:37.330438 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:37.330620 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:37.330620 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:37.330621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:37.330724 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:37.330778 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:37.330914 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 08:56:37.330994 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":76,"threads_started":1}
I20260619 08:56:37.331162 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:37.324725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44545 (local address 127.0.0.1:47448)
0619 08:56:37.324881 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:37.324893 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:37.324962 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:37.325084 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:37.325085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:37.325384 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:37.325438 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:37.325440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:37.326060 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:37.326061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:37.326600 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:37.326602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:37.326913 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:37.326914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:37.326980 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:37.327474 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:37.327478 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:37.328809 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:37.330092 (+ 1283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:37.330098 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:37.330100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:37.330307 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:37.330513 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:37.330514 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:37.330514 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:37.330567 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:37.330779 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:37.330787 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:37.330832 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 08:56:37.330887 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 08:56:37.331058 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":72,"thread_start_us":77,"threads_started":1}
I20260619 08:56:37.333179 7909 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47448:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 08:56:37.333377 7909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:56:37.334695 8309 tablet_service.cc:1467] Processing CreateTablet for tablet afe122c86e644609aa7835682ef5f917 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cb7283974d384af886fb9265a0600062]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 08:56:37.334679 8046 tablet_service.cc:1467] Processing CreateTablet for tablet afe122c86e644609aa7835682ef5f917 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cb7283974d384af886fb9265a0600062]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 08:56:37.334707 8186 tablet_service.cc:1467] Processing CreateTablet for tablet afe122c86e644609aa7835682ef5f917 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cb7283974d384af886fb9265a0600062]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 08:56:37.334836 8046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afe122c86e644609aa7835682ef5f917. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:37.334836 8186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afe122c86e644609aa7835682ef5f917. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:37.334836 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afe122c86e644609aa7835682ef5f917. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:37.335944 8420 tablet_bootstrap.cc:492] T afe122c86e644609aa7835682ef5f917 P eac8f4fc9d344cee8cf11be42b0c0b3b: Bootstrap starting.
I20260619 08:56:37.336297 8421 tablet_bootstrap.cc:492] T afe122c86e644609aa7835682ef5f917 P a770824c003e422f8fd8ef3d458332ee: Bootstrap starting.
I20260619 08:56:37.336352 8420 tablet_bootstrap.cc:654] T afe122c86e644609aa7835682ef5f917 P eac8f4fc9d344cee8cf11be42b0c0b3b: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:37.336361 8423 tablet_bootstrap.cc:492] T afe122c86e644609aa7835682ef5f917 P 33cbeb4cbd1840b5a05436a1aef6bf2c: Bootstrap starting.
I20260619 08:56:37.336628 8421 tablet_bootstrap.cc:654] T afe122c86e644609aa7835682ef5f917 P a770824c003e422f8fd8ef3d458332ee: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:37.336709 8423 tablet_bootstrap.cc:654] T afe122c86e644609aa7835682ef5f917 P 33cbeb4cbd1840b5a05436a1aef6bf2c: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:37.337138 8421 tablet_bootstrap.cc:492] T afe122c86e644609aa7835682ef5f917 P a770824c003e422f8fd8ef3d458332ee: No bootstrap required, opened a new log
I20260619 08:56:37.337189 8421 ts_tablet_manager.cc:1397] T afe122c86e644609aa7835682ef5f917 P a770824c003e422f8fd8ef3d458332ee: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 08:56:37.337347 8420 tablet_bootstrap.cc:492] T afe122c86e644609aa7835682ef5f917 P eac8f4fc9d344cee8cf11be42b0c0b3b: No bootstrap required, opened a new log
I20260619 08:56:37.337324 8421 raft_consensus.cc:348] T afe122c86e644609aa7835682ef5f917 P a770824c003e422f8fd8ef3d458332ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } }
I20260619 08:56:37.337388 8421 raft_consensus.cc:374] T afe122c86e644609aa7835682ef5f917 P a770824c003e422f8fd8ef3d458332ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:37.337393 8420 ts_tablet_manager.cc:1397] T afe122c86e644609aa7835682ef5f917 P eac8f4fc9d344cee8cf11be42b0c0b3b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 08:56:37.337417 8421 raft_consensus.cc:729] T afe122c86e644609aa7835682ef5f917 P a770824c003e422f8fd8ef3d458332ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a770824c003e422f8fd8ef3d458332ee, State: Initialized, Role: FOLLOWER
I20260619 08:56:37.337466 8421 consensus_queue.cc:260] T afe122c86e644609aa7835682ef5f917 P a770824c003e422f8fd8ef3d458332ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } }
I20260619 08:56:37.337474 8423 tablet_bootstrap.cc:492] T afe122c86e644609aa7835682ef5f917 P 33cbeb4cbd1840b5a05436a1aef6bf2c: No bootstrap required, opened a new log
I20260619 08:56:37.337513 8423 ts_tablet_manager.cc:1397] T afe122c86e644609aa7835682ef5f917 P 33cbeb4cbd1840b5a05436a1aef6bf2c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 08:56:37.337548 8421 ts_tablet_manager.cc:1428] T afe122c86e644609aa7835682ef5f917 P a770824c003e422f8fd8ef3d458332ee: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:37.337522 8420 raft_consensus.cc:348] T afe122c86e644609aa7835682ef5f917 P eac8f4fc9d344cee8cf11be42b0c0b3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } }
I20260619 08:56:37.337579 8420 raft_consensus.cc:374] T afe122c86e644609aa7835682ef5f917 P eac8f4fc9d344cee8cf11be42b0c0b3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:37.337597 8420 raft_consensus.cc:729] T afe122c86e644609aa7835682ef5f917 P eac8f4fc9d344cee8cf11be42b0c0b3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eac8f4fc9d344cee8cf11be42b0c0b3b, State: Initialized, Role: FOLLOWER
I20260619 08:56:37.337579 8438 tablet_replica.cc:439] T afe122c86e644609aa7835682ef5f917 P a770824c003e422f8fd8ef3d458332ee: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } } }
I20260619 08:56:37.337636 8420 consensus_queue.cc:260] T afe122c86e644609aa7835682ef5f917 P eac8f4fc9d344cee8cf11be42b0c0b3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } }
I20260619 08:56:37.337667 8438 tablet_replica.cc:442] T afe122c86e644609aa7835682ef5f917 P a770824c003e422f8fd8ef3d458332ee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:56:37.337638 8423 raft_consensus.cc:348] T afe122c86e644609aa7835682ef5f917 P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } }
I20260619 08:56:37.337704 8420 ts_tablet_manager.cc:1428] T afe122c86e644609aa7835682ef5f917 P eac8f4fc9d344cee8cf11be42b0c0b3b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:37.337706 8423 raft_consensus.cc:374] T afe122c86e644609aa7835682ef5f917 P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:37.337728 8423 raft_consensus.cc:729] T afe122c86e644609aa7835682ef5f917 P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33cbeb4cbd1840b5a05436a1aef6bf2c, State: Initialized, Role: FOLLOWER
I20260619 08:56:37.337762 8423 consensus_queue.cc:260] T afe122c86e644609aa7835682ef5f917 P 33cbeb4cbd1840b5a05436a1aef6bf2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } }
I20260619 08:56:37.337837 8423 ts_tablet_manager.cc:1428] T afe122c86e644609aa7835682ef5f917 P 33cbeb4cbd1840b5a05436a1aef6bf2c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:37.337811 8426 tablet_replica.cc:439] T afe122c86e644609aa7835682ef5f917 P eac8f4fc9d344cee8cf11be42b0c0b3b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } } }
I20260619 08:56:37.337872 8426 tablet_replica.cc:442] T afe122c86e644609aa7835682ef5f917 P eac8f4fc9d344cee8cf11be42b0c0b3b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:56:37.337972 8428 tablet_replica.cc:439] T afe122c86e644609aa7835682ef5f917 P 33cbeb4cbd1840b5a05436a1aef6bf2c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } } }
I20260619 08:56:37.338032 8428 tablet_replica.cc:442] T afe122c86e644609aa7835682ef5f917 P 33cbeb4cbd1840b5a05436a1aef6bf2c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:56:37.543582 8428 raft_consensus.cc:482] T afe122c86e644609aa7835682ef5f917 P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:56:37.543684 8428 raft_consensus.cc:504] T afe122c86e644609aa7835682ef5f917 P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } }
I20260619 08:56:37.543829 8428 leader_election.cc:288] T afe122c86e644609aa7835682ef5f917 P 33cbeb4cbd1840b5a05436a1aef6bf2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a770824c003e422f8fd8ef3d458332ee (127.0.98.2:36709), eac8f4fc9d344cee8cf11be42b0c0b3b (127.0.98.3:40723)
I20260619 08:56:37.544060 8342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afe122c86e644609aa7835682ef5f917" candidate_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" is_pre_election: true
I20260619 08:56:37.544107 8206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afe122c86e644609aa7835682ef5f917" candidate_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a770824c003e422f8fd8ef3d458332ee" is_pre_election: true
I20260619 08:56:37.544199 8206 raft_consensus.cc:2454] T afe122c86e644609aa7835682ef5f917 P a770824c003e422f8fd8ef3d458332ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33cbeb4cbd1840b5a05436a1aef6bf2c in term 0.
I20260619 08:56:37.544214 8342 raft_consensus.cc:2454] T afe122c86e644609aa7835682ef5f917 P eac8f4fc9d344cee8cf11be42b0c0b3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33cbeb4cbd1840b5a05436a1aef6bf2c in term 0.
I20260619 08:56:37.544354 7999 leader_election.cc:302] T afe122c86e644609aa7835682ef5f917 P 33cbeb4cbd1840b5a05436a1aef6bf2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33cbeb4cbd1840b5a05436a1aef6bf2c, eac8f4fc9d344cee8cf11be42b0c0b3b; no voters:
I20260619 08:56:37.544476 8428 raft_consensus.cc:2781] T afe122c86e644609aa7835682ef5f917 P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:56:37.544531 8428 raft_consensus.cc:482] T afe122c86e644609aa7835682ef5f917 P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:56:37.544552 8428 raft_consensus.cc:3037] T afe122c86e644609aa7835682ef5f917 P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:37.545193 8428 raft_consensus.cc:504] T afe122c86e644609aa7835682ef5f917 P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } }
I20260619 08:56:37.545320 8428 leader_election.cc:288] T afe122c86e644609aa7835682ef5f917 P 33cbeb4cbd1840b5a05436a1aef6bf2c [CANDIDATE]: Term 1 election: Requested vote from peers a770824c003e422f8fd8ef3d458332ee (127.0.98.2:36709), eac8f4fc9d344cee8cf11be42b0c0b3b (127.0.98.3:40723)
I20260619 08:56:37.545460 8206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afe122c86e644609aa7835682ef5f917" candidate_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a770824c003e422f8fd8ef3d458332ee"
I20260619 08:56:37.545456 8342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afe122c86e644609aa7835682ef5f917" candidate_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b"
I20260619 08:56:37.545531 8206 raft_consensus.cc:3037] T afe122c86e644609aa7835682ef5f917 P a770824c003e422f8fd8ef3d458332ee [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:37.545529 8342 raft_consensus.cc:3037] T afe122c86e644609aa7835682ef5f917 P eac8f4fc9d344cee8cf11be42b0c0b3b [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:37.546104 8206 raft_consensus.cc:2454] T afe122c86e644609aa7835682ef5f917 P a770824c003e422f8fd8ef3d458332ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33cbeb4cbd1840b5a05436a1aef6bf2c in term 1.
I20260619 08:56:37.546190 8342 raft_consensus.cc:2454] T afe122c86e644609aa7835682ef5f917 P eac8f4fc9d344cee8cf11be42b0c0b3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33cbeb4cbd1840b5a05436a1aef6bf2c in term 1.
I20260619 08:56:37.546291 7997 leader_election.cc:302] T afe122c86e644609aa7835682ef5f917 P 33cbeb4cbd1840b5a05436a1aef6bf2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33cbeb4cbd1840b5a05436a1aef6bf2c, a770824c003e422f8fd8ef3d458332ee; no voters:
I20260619 08:56:37.546381 8428 raft_consensus.cc:2781] T afe122c86e644609aa7835682ef5f917 P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:37.546443 8428 raft_consensus.cc:686] T afe122c86e644609aa7835682ef5f917 P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 1 LEADER]: Becoming Leader. State: Replica: 33cbeb4cbd1840b5a05436a1aef6bf2c, State: Running, Role: LEADER
I20260619 08:56:37.546500 8428 consensus_queue.cc:237] T afe122c86e644609aa7835682ef5f917 P 33cbeb4cbd1840b5a05436a1aef6bf2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } }
I20260619 08:56:37.546797 8451 tablet_replica.cc:439] T afe122c86e644609aa7835682ef5f917 P 33cbeb4cbd1840b5a05436a1aef6bf2c: TxnStatusTablet state changed. Reason: New leader 33cbeb4cbd1840b5a05436a1aef6bf2c. Latest consensus state: current_term: 1 leader_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } } }
I20260619 08:56:37.546861 8451 tablet_replica.cc:442] T afe122c86e644609aa7835682ef5f917 P 33cbeb4cbd1840b5a05436a1aef6bf2c: This TxnStatusTablet replica's current role is: LEADER
I20260619 08:56:37.547021 8464 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 08:56:37.547130 8464 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 08:56:37.547107 7908 catalog_manager.cc:5495] T afe122c86e644609aa7835682ef5f917 P 33cbeb4cbd1840b5a05436a1aef6bf2c reported cstate change: term changed from 0 to 1, leader changed from <none> to 33cbeb4cbd1840b5a05436a1aef6bf2c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } health_report { overall_health: UNKNOWN } } }
I20260619 08:56:37.565848 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:37.563082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60338 (local address 127.0.98.1:45395)
0619 08:56:37.563208 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:37.563210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:37.563218 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:56:37.563250 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:37.563251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:37.563274 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:37.563338 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:56:37.563465 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:37.563962 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:37.565096 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:37.565316 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:37.565380 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:37.565546 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:56:37.565642 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:56:37.565695 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:37.565752 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 08:56:37.565769 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":56,"threads_started":1}
I20260619 08:56:37.565883 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:37.563010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45395 (local address 127.0.0.1:60338)
0619 08:56:37.563082 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:37.563094 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:37.563143 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:37.563353 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:37.563354 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:37.563355 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 08:56:37.563407 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:37.563410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:37.564027 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:37.564028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:37.565006 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:37.565009 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:37.565387 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:37.565388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:37.565444 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:37.565463 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 08:56:37.565693 (+ 230us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 08:56:37.565698 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 08:56:37.565749 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 08:56:37.565799 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 08:56:37.567205 8342 raft_consensus.cc:1261] T afe122c86e644609aa7835682ef5f917 P eac8f4fc9d344cee8cf11be42b0c0b3b [term 1 FOLLOWER]: Refusing update from remote peer 33cbeb4cbd1840b5a05436a1aef6bf2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:56:37.567261 8206 raft_consensus.cc:1261] T afe122c86e644609aa7835682ef5f917 P a770824c003e422f8fd8ef3d458332ee [term 1 FOLLOWER]: Refusing update from remote peer 33cbeb4cbd1840b5a05436a1aef6bf2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:56:37.567464 8451 consensus_queue.cc:1035] T afe122c86e644609aa7835682ef5f917 P 33cbeb4cbd1840b5a05436a1aef6bf2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:56:37.567600 8428 consensus_queue.cc:1035] T afe122c86e644609aa7835682ef5f917 P 33cbeb4cbd1840b5a05436a1aef6bf2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:56:37.568241 8426 tablet_replica.cc:439] T afe122c86e644609aa7835682ef5f917 P eac8f4fc9d344cee8cf11be42b0c0b3b: TxnStatusTablet state changed. Reason: New leader 33cbeb4cbd1840b5a05436a1aef6bf2c. Latest consensus state: current_term: 1 leader_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } } }
I20260619 08:56:37.568344 8426 tablet_replica.cc:442] T afe122c86e644609aa7835682ef5f917 P eac8f4fc9d344cee8cf11be42b0c0b3b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:56:37.568464 8438 tablet_replica.cc:439] T afe122c86e644609aa7835682ef5f917 P a770824c003e422f8fd8ef3d458332ee: TxnStatusTablet state changed. Reason: New leader 33cbeb4cbd1840b5a05436a1aef6bf2c. Latest consensus state: current_term: 1 leader_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } } }
I20260619 08:56:37.568602 8438 tablet_replica.cc:442] T afe122c86e644609aa7835682ef5f917 P a770824c003e422f8fd8ef3d458332ee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:56:37.569525 8451 tablet_replica.cc:439] T afe122c86e644609aa7835682ef5f917 P 33cbeb4cbd1840b5a05436a1aef6bf2c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } } }
I20260619 08:56:37.569609 8451 tablet_replica.cc:442] T afe122c86e644609aa7835682ef5f917 P 33cbeb4cbd1840b5a05436a1aef6bf2c: This TxnStatusTablet replica's current role is: LEADER
I20260619 08:56:37.569651 8451 tablet_replica.cc:439] T afe122c86e644609aa7835682ef5f917 P 33cbeb4cbd1840b5a05436a1aef6bf2c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } } }
I20260619 08:56:37.569689 8451 tablet_replica.cc:442] T afe122c86e644609aa7835682ef5f917 P 33cbeb4cbd1840b5a05436a1aef6bf2c: This TxnStatusTablet replica's current role is: LEADER
I20260619 08:56:37.569638 8426 tablet_replica.cc:439] T afe122c86e644609aa7835682ef5f917 P eac8f4fc9d344cee8cf11be42b0c0b3b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } } }
I20260619 08:56:37.569711 8426 tablet_replica.cc:442] T afe122c86e644609aa7835682ef5f917 P eac8f4fc9d344cee8cf11be42b0c0b3b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:56:37.569778 8470 mvcc.cc:205] Tried to move back new op lower bound from 7298496092431728640 to 7298496092349374464. Current Snapshot: MvccSnapshot[applied={T|T < 7298496092431728640}]
I20260619 08:56:37.569867 8438 tablet_replica.cc:439] T afe122c86e644609aa7835682ef5f917 P a770824c003e422f8fd8ef3d458332ee: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } } }
I20260619 08:56:37.569928 8438 tablet_replica.cc:442] T afe122c86e644609aa7835682ef5f917 P a770824c003e422f8fd8ef3d458332ee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 08:56:37.570128 8468 mvcc.cc:205] Tried to move back new op lower bound from 7298496092431728640 to 7298496092349374464. Current Snapshot: MvccSnapshot[applied={T|T < 7298496092431728640}]
I20260619 08:56:37.574539 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:37.572022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60354 (local address 127.0.98.1:45395)
0619 08:56:37.572062 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:37.572063 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:37.572120 (+ 57us) server_negotiation.cc:407] Connection header received
0619 08:56:37.572182 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:37.572182 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:37.572190 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:37.572238 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:56:37.572364 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:37.572823 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:37.573919 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:37.574125 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:37.574180 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:37.574337 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:56:37.574372 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:56:37.574405 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:37.574470 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 08:56:37.574486 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 08:56:37.579808 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:37.576972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54523 (local address 127.0.98.1:45395)
0619 08:56:37.577006 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:37.577008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:37.577122 (+ 114us) server_negotiation.cc:407] Connection header received
0619 08:56:37.577179 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:37.577180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:37.577187 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:37.577225 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:56:37.577350 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:37.578137 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:37.579229 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:37.579397 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:37.579447 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:37.579609 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:56:37.579630 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:56:37.579660 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:37.579722 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 08:56:37.579743 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":16}
I20260619 08:56:37.579908 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:37.576917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45395 (local address 127.0.98.1:54523)
0619 08:56:37.577063 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:37.577073 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:37.577126 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:37.577242 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:37.577243 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:37.577254 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 08:56:37.577306 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:37.577307 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:37.578216 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:37.578217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:37.579150 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:37.579152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:37.579462 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:37.579463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:37.579524 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:37.579539 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 08:56:37.579674 (+ 135us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 08:56:37.579675 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 08:56:37.579719 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 08:56:37.579767 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":4,"thread_start_us":69,"threads_started":1}
I20260619 08:56:37.581974 8206 raft_consensus.cc:1261] T 79004176209441dbba47b63cfe32927b P a770824c003e422f8fd8ef3d458332ee [term 1 FOLLOWER]: Refusing update from remote peer 33cbeb4cbd1840b5a05436a1aef6bf2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:56:37.581974 8342 raft_consensus.cc:1261] T 79004176209441dbba47b63cfe32927b P eac8f4fc9d344cee8cf11be42b0c0b3b [term 1 FOLLOWER]: Refusing update from remote peer 33cbeb4cbd1840b5a05436a1aef6bf2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:56:37.582177 8481 consensus_queue.cc:1035] T 79004176209441dbba47b63cfe32927b P 33cbeb4cbd1840b5a05436a1aef6bf2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:56:37.582295 8451 consensus_queue.cc:1035] T 79004176209441dbba47b63cfe32927b P 33cbeb4cbd1840b5a05436a1aef6bf2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:56:37.592929 7909 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:56:37.593020 7909 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:56:37.593573 7909 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57ff80e31aa34b1d880dacffdefa69ab: Sending DeleteTablet for 3 replicas of tablet 79004176209441dbba47b63cfe32927b
I20260619 08:56:37.593817 8186 tablet_service.cc:1514] Processing DeleteTablet for tablet 79004176209441dbba47b63cfe32927b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33474
I20260619 08:56:37.593843 8046 tablet_service.cc:1514] Processing DeleteTablet for tablet 79004176209441dbba47b63cfe32927b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60324
I20260619 08:56:37.593842 8309 tablet_service.cc:1514] Processing DeleteTablet for tablet 79004176209441dbba47b63cfe32927b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44430
I20260619 08:56:37.594115 8486 tablet_replica.cc:330] T 79004176209441dbba47b63cfe32927b P 33cbeb4cbd1840b5a05436a1aef6bf2c: stopping tablet replica
I20260619 08:56:37.596055 8487 tablet_replica.cc:330] T 79004176209441dbba47b63cfe32927b P eac8f4fc9d344cee8cf11be42b0c0b3b: stopping tablet replica
I20260619 08:56:37.596172 8487 raft_consensus.cc:2229] T 79004176209441dbba47b63cfe32927b P eac8f4fc9d344cee8cf11be42b0c0b3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:56:37.596226 8487 raft_consensus.cc:2258] T 79004176209441dbba47b63cfe32927b P eac8f4fc9d344cee8cf11be42b0c0b3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:37.599717 8487 ts_tablet_manager.cc:1905] T 79004176209441dbba47b63cfe32927b P eac8f4fc9d344cee8cf11be42b0c0b3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:37.601475 8487 ts_tablet_manager.cc:1918] T 79004176209441dbba47b63cfe32927b P eac8f4fc9d344cee8cf11be42b0c0b3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 08:56:37.601527 8487 log.cc:1196] T 79004176209441dbba47b63cfe32927b P eac8f4fc9d344cee8cf11be42b0c0b3b: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-2/wal/wals/79004176209441dbba47b63cfe32927b
I20260619 08:56:37.601715 8487 ts_tablet_manager.cc:1939] T 79004176209441dbba47b63cfe32927b P eac8f4fc9d344cee8cf11be42b0c0b3b: Deleting consensus metadata
I20260619 08:56:37.602023 7895 catalog_manager.cc:4841] TS eac8f4fc9d344cee8cf11be42b0c0b3b (127.0.98.3:40723): tablet 79004176209441dbba47b63cfe32927b (table test-table [id=186eced15a124c6db180fc2d742fe8cc]) successfully deleted
I20260619 08:56:37.604141 8486 raft_consensus.cc:2229] T 79004176209441dbba47b63cfe32927b P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:56:37.604189 8485 tablet_replica.cc:330] T 79004176209441dbba47b63cfe32927b P a770824c003e422f8fd8ef3d458332ee: stopping tablet replica
I20260619 08:56:37.604249 8485 raft_consensus.cc:2229] T 79004176209441dbba47b63cfe32927b P a770824c003e422f8fd8ef3d458332ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:56:37.604274 8486 raft_consensus.cc:2258] T 79004176209441dbba47b63cfe32927b P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:37.604310 8485 raft_consensus.cc:2258] T 79004176209441dbba47b63cfe32927b P a770824c003e422f8fd8ef3d458332ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:37.604569 8485 ts_tablet_manager.cc:1905] T 79004176209441dbba47b63cfe32927b P a770824c003e422f8fd8ef3d458332ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:37.604570 8486 ts_tablet_manager.cc:1905] T 79004176209441dbba47b63cfe32927b P 33cbeb4cbd1840b5a05436a1aef6bf2c: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 08:56:37 dist-test-slave-h4g5 krb5kdc[7864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260619 08:56:37.606249 8486 ts_tablet_manager.cc:1918] T 79004176209441dbba47b63cfe32927b P 33cbeb4cbd1840b5a05436a1aef6bf2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 08:56:37.606249 8485 ts_tablet_manager.cc:1918] T 79004176209441dbba47b63cfe32927b P a770824c003e422f8fd8ef3d458332ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 08:56:37.606305 8486 log.cc:1196] T 79004176209441dbba47b63cfe32927b P 33cbeb4cbd1840b5a05436a1aef6bf2c: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-0/wal/wals/79004176209441dbba47b63cfe32927b
I20260619 08:56:37.606305 8485 log.cc:1196] T 79004176209441dbba47b63cfe32927b P a770824c003e422f8fd8ef3d458332ee: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-1/wal/wals/79004176209441dbba47b63cfe32927b
I20260619 08:56:37.606504 8486 ts_tablet_manager.cc:1939] T 79004176209441dbba47b63cfe32927b P 33cbeb4cbd1840b5a05436a1aef6bf2c: Deleting consensus metadata
I20260619 08:56:37.606504 8485 ts_tablet_manager.cc:1939] T 79004176209441dbba47b63cfe32927b P a770824c003e422f8fd8ef3d458332ee: Deleting consensus metadata
I20260619 08:56:37.606827 7895 catalog_manager.cc:4841] TS 33cbeb4cbd1840b5a05436a1aef6bf2c (127.0.98.1:45395): tablet 79004176209441dbba47b63cfe32927b (table test-table [id=186eced15a124c6db180fc2d742fe8cc]) successfully deleted
I20260619 08:56:37.606920 7893 catalog_manager.cc:4841] TS a770824c003e422f8fd8ef3d458332ee (127.0.98.2:36709): tablet 79004176209441dbba47b63cfe32927b (table test-table [id=186eced15a124c6db180fc2d742fe8cc]) successfully deleted
Jun 19 08:56:37 dist-test-slave-h4g5 krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:37 dist-test-slave-h4g5 krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 08:56:37.619766 8459 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:37.614443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47458 (local address 127.0.98.62:44545)
0619 08:56:37.614504 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:37.614506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:37.614634 (+ 128us) server_negotiation.cc:407] Connection header received
0619 08:56:37.614695 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:37.614696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:37.614707 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:37.614759 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:37.615286 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:37.615788 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:37.616545 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:37.616797 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:37.616853 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:37.619675 (+ 2822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47458: BlockingRecv error: recv got EOF from 127.0.0.1:47458 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Jun 19 08:56:37 dist-test-slave-h4g5 krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 08:56:37.627004 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:37.620513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47474 (local address 127.0.98.62:44545)
0619 08:56:37.620547 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:37.620549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:37.620655 (+ 106us) server_negotiation.cc:407] Connection header received
0619 08:56:37.620723 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:37.620724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:37.620734 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:37.620776 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:37.621182 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:37.621639 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:37.622285 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:37.622467 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:37.622520 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:37.624791 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:37.624792 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:37.624793 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:37.624799 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:37.625967 (+ 1168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:37.626321 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:37.626321 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:37.626322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:37.626364 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:37.626539 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:37.626539 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:37.626540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:37.626627 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:37.626712 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:37.626824 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 08:56:37.626901 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 08:56:37.628563 7909 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:56:37.628715 7909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:56:37.629990 8046 tablet_service.cc:1467] Processing CreateTablet for tablet 7c5ebe0596ac4774b9494c2e4d3680d6 (DEFAULT_TABLE table=test-table [id=e48aa9db5a494599bf976ff82b93a97d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:37.630069 8309 tablet_service.cc:1467] Processing CreateTablet for tablet 7c5ebe0596ac4774b9494c2e4d3680d6 (DEFAULT_TABLE table=test-table [id=e48aa9db5a494599bf976ff82b93a97d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:37.630119 8046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c5ebe0596ac4774b9494c2e4d3680d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:37.630107 8186 tablet_service.cc:1467] Processing CreateTablet for tablet 7c5ebe0596ac4774b9494c2e4d3680d6 (DEFAULT_TABLE table=test-table [id=e48aa9db5a494599bf976ff82b93a97d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:37.630182 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c5ebe0596ac4774b9494c2e4d3680d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:37.630214 8186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c5ebe0596ac4774b9494c2e4d3680d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:37.631512 8421 tablet_bootstrap.cc:492] T 7c5ebe0596ac4774b9494c2e4d3680d6 P a770824c003e422f8fd8ef3d458332ee: Bootstrap starting.
I20260619 08:56:37.631775 8423 tablet_bootstrap.cc:492] T 7c5ebe0596ac4774b9494c2e4d3680d6 P 33cbeb4cbd1840b5a05436a1aef6bf2c: Bootstrap starting.
I20260619 08:56:37.631826 8421 tablet_bootstrap.cc:654] T 7c5ebe0596ac4774b9494c2e4d3680d6 P a770824c003e422f8fd8ef3d458332ee: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:37.632051 8420 tablet_bootstrap.cc:492] T 7c5ebe0596ac4774b9494c2e4d3680d6 P eac8f4fc9d344cee8cf11be42b0c0b3b: Bootstrap starting.
I20260619 08:56:37.632318 8423 tablet_bootstrap.cc:654] T 7c5ebe0596ac4774b9494c2e4d3680d6 P 33cbeb4cbd1840b5a05436a1aef6bf2c: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:37.632411 8420 tablet_bootstrap.cc:654] T 7c5ebe0596ac4774b9494c2e4d3680d6 P eac8f4fc9d344cee8cf11be42b0c0b3b: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:37.632921 8423 tablet_bootstrap.cc:492] T 7c5ebe0596ac4774b9494c2e4d3680d6 P 33cbeb4cbd1840b5a05436a1aef6bf2c: No bootstrap required, opened a new log
I20260619 08:56:37.632979 8423 ts_tablet_manager.cc:1397] T 7c5ebe0596ac4774b9494c2e4d3680d6 P 33cbeb4cbd1840b5a05436a1aef6bf2c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 08:56:37.633003 8421 tablet_bootstrap.cc:492] T 7c5ebe0596ac4774b9494c2e4d3680d6 P a770824c003e422f8fd8ef3d458332ee: No bootstrap required, opened a new log
I20260619 08:56:37.633049 8421 ts_tablet_manager.cc:1397] T 7c5ebe0596ac4774b9494c2e4d3680d6 P a770824c003e422f8fd8ef3d458332ee: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 08:56:37.633150 8421 raft_consensus.cc:348] T 7c5ebe0596ac4774b9494c2e4d3680d6 P a770824c003e422f8fd8ef3d458332ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } }
I20260619 08:56:37.633208 8421 raft_consensus.cc:374] T 7c5ebe0596ac4774b9494c2e4d3680d6 P a770824c003e422f8fd8ef3d458332ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:37.633147 8423 raft_consensus.cc:348] T 7c5ebe0596ac4774b9494c2e4d3680d6 P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } }
I20260619 08:56:37.633220 8421 raft_consensus.cc:729] T 7c5ebe0596ac4774b9494c2e4d3680d6 P a770824c003e422f8fd8ef3d458332ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a770824c003e422f8fd8ef3d458332ee, State: Initialized, Role: FOLLOWER
I20260619 08:56:37.633226 8423 raft_consensus.cc:374] T 7c5ebe0596ac4774b9494c2e4d3680d6 P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:37.633247 8423 raft_consensus.cc:729] T 7c5ebe0596ac4774b9494c2e4d3680d6 P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33cbeb4cbd1840b5a05436a1aef6bf2c, State: Initialized, Role: FOLLOWER
I20260619 08:56:37.633255 8421 consensus_queue.cc:260] T 7c5ebe0596ac4774b9494c2e4d3680d6 P a770824c003e422f8fd8ef3d458332ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } }
I20260619 08:56:37.633286 8423 consensus_queue.cc:260] T 7c5ebe0596ac4774b9494c2e4d3680d6 P 33cbeb4cbd1840b5a05436a1aef6bf2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } }
I20260619 08:56:37.633337 8421 ts_tablet_manager.cc:1428] T 7c5ebe0596ac4774b9494c2e4d3680d6 P a770824c003e422f8fd8ef3d458332ee: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:37.633387 8423 ts_tablet_manager.cc:1428] T 7c5ebe0596ac4774b9494c2e4d3680d6 P 33cbeb4cbd1840b5a05436a1aef6bf2c: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260619 08:56:37.633915 8420 tablet_bootstrap.cc:492] T 7c5ebe0596ac4774b9494c2e4d3680d6 P eac8f4fc9d344cee8cf11be42b0c0b3b: No bootstrap required, opened a new log
I20260619 08:56:37.633962 8420 ts_tablet_manager.cc:1397] T 7c5ebe0596ac4774b9494c2e4d3680d6 P eac8f4fc9d344cee8cf11be42b0c0b3b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 08:56:37.634102 8420 raft_consensus.cc:348] T 7c5ebe0596ac4774b9494c2e4d3680d6 P eac8f4fc9d344cee8cf11be42b0c0b3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } }
I20260619 08:56:37.634162 8420 raft_consensus.cc:374] T 7c5ebe0596ac4774b9494c2e4d3680d6 P eac8f4fc9d344cee8cf11be42b0c0b3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:37.634181 8420 raft_consensus.cc:729] T 7c5ebe0596ac4774b9494c2e4d3680d6 P eac8f4fc9d344cee8cf11be42b0c0b3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eac8f4fc9d344cee8cf11be42b0c0b3b, State: Initialized, Role: FOLLOWER
I20260619 08:56:37.634217 8420 consensus_queue.cc:260] T 7c5ebe0596ac4774b9494c2e4d3680d6 P eac8f4fc9d344cee8cf11be42b0c0b3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } }
I20260619 08:56:37.634291 8420 ts_tablet_manager.cc:1428] T 7c5ebe0596ac4774b9494c2e4d3680d6 P eac8f4fc9d344cee8cf11be42b0c0b3b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:37.827638 8438 raft_consensus.cc:482] T 7c5ebe0596ac4774b9494c2e4d3680d6 P a770824c003e422f8fd8ef3d458332ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:56:37.827752 8438 raft_consensus.cc:504] T 7c5ebe0596ac4774b9494c2e4d3680d6 P a770824c003e422f8fd8ef3d458332ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } }
I20260619 08:56:37.827900 8438 leader_election.cc:288] T 7c5ebe0596ac4774b9494c2e4d3680d6 P a770824c003e422f8fd8ef3d458332ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33cbeb4cbd1840b5a05436a1aef6bf2c (127.0.98.1:45395), eac8f4fc9d344cee8cf11be42b0c0b3b (127.0.98.3:40723)
I20260619 08:56:37.828169 8050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c5ebe0596ac4774b9494c2e4d3680d6" candidate_uuid: "a770824c003e422f8fd8ef3d458332ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" is_pre_election: true
I20260619 08:56:37.828193 8342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c5ebe0596ac4774b9494c2e4d3680d6" candidate_uuid: "a770824c003e422f8fd8ef3d458332ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" is_pre_election: true
I20260619 08:56:37.828274 8050 raft_consensus.cc:2454] T 7c5ebe0596ac4774b9494c2e4d3680d6 P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a770824c003e422f8fd8ef3d458332ee in term 0.
I20260619 08:56:37.828277 8342 raft_consensus.cc:2454] T 7c5ebe0596ac4774b9494c2e4d3680d6 P eac8f4fc9d344cee8cf11be42b0c0b3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a770824c003e422f8fd8ef3d458332ee in term 0.
I20260619 08:56:37.828428 8139 leader_election.cc:302] T 7c5ebe0596ac4774b9494c2e4d3680d6 P a770824c003e422f8fd8ef3d458332ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a770824c003e422f8fd8ef3d458332ee, eac8f4fc9d344cee8cf11be42b0c0b3b; no voters:
I20260619 08:56:37.828534 8438 raft_consensus.cc:2781] T 7c5ebe0596ac4774b9494c2e4d3680d6 P a770824c003e422f8fd8ef3d458332ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:56:37.828586 8438 raft_consensus.cc:482] T 7c5ebe0596ac4774b9494c2e4d3680d6 P a770824c003e422f8fd8ef3d458332ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:56:37.828610 8438 raft_consensus.cc:3037] T 7c5ebe0596ac4774b9494c2e4d3680d6 P a770824c003e422f8fd8ef3d458332ee [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:37.829306 8438 raft_consensus.cc:504] T 7c5ebe0596ac4774b9494c2e4d3680d6 P a770824c003e422f8fd8ef3d458332ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } }
I20260619 08:56:37.829407 8438 leader_election.cc:288] T 7c5ebe0596ac4774b9494c2e4d3680d6 P a770824c003e422f8fd8ef3d458332ee [CANDIDATE]: Term 1 election: Requested vote from peers 33cbeb4cbd1840b5a05436a1aef6bf2c (127.0.98.1:45395), eac8f4fc9d344cee8cf11be42b0c0b3b (127.0.98.3:40723)
I20260619 08:56:37.829618 8050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c5ebe0596ac4774b9494c2e4d3680d6" candidate_uuid: "a770824c003e422f8fd8ef3d458332ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c"
I20260619 08:56:37.829627 8342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c5ebe0596ac4774b9494c2e4d3680d6" candidate_uuid: "a770824c003e422f8fd8ef3d458332ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b"
I20260619 08:56:37.829684 8050 raft_consensus.cc:3037] T 7c5ebe0596ac4774b9494c2e4d3680d6 P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:37.829694 8342 raft_consensus.cc:3037] T 7c5ebe0596ac4774b9494c2e4d3680d6 P eac8f4fc9d344cee8cf11be42b0c0b3b [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:37.830241 8050 raft_consensus.cc:2454] T 7c5ebe0596ac4774b9494c2e4d3680d6 P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a770824c003e422f8fd8ef3d458332ee in term 1.
I20260619 08:56:37.830312 8342 raft_consensus.cc:2454] T 7c5ebe0596ac4774b9494c2e4d3680d6 P eac8f4fc9d344cee8cf11be42b0c0b3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a770824c003e422f8fd8ef3d458332ee in term 1.
I20260619 08:56:37.830433 8139 leader_election.cc:302] T 7c5ebe0596ac4774b9494c2e4d3680d6 P a770824c003e422f8fd8ef3d458332ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33cbeb4cbd1840b5a05436a1aef6bf2c, a770824c003e422f8fd8ef3d458332ee; no voters:
I20260619 08:56:37.830529 8438 raft_consensus.cc:2781] T 7c5ebe0596ac4774b9494c2e4d3680d6 P a770824c003e422f8fd8ef3d458332ee [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:37.830643 8438 raft_consensus.cc:686] T 7c5ebe0596ac4774b9494c2e4d3680d6 P a770824c003e422f8fd8ef3d458332ee [term 1 LEADER]: Becoming Leader. State: Replica: a770824c003e422f8fd8ef3d458332ee, State: Running, Role: LEADER
I20260619 08:56:37.830701 8438 consensus_queue.cc:237] T 7c5ebe0596ac4774b9494c2e4d3680d6 P a770824c003e422f8fd8ef3d458332ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } }
I20260619 08:56:37.831290 7909 catalog_manager.cc:5495] T 7c5ebe0596ac4774b9494c2e4d3680d6 P a770824c003e422f8fd8ef3d458332ee reported cstate change: term changed from 0 to 1, leader changed from <none> to a770824c003e422f8fd8ef3d458332ee (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a770824c003e422f8fd8ef3d458332ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } health_report { overall_health: UNKNOWN } } }
I20260619 08:56:37.859624 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:37.857497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33494 (local address 127.0.98.2:36709)
0619 08:56:37.857605 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:37.857607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:37.857616 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:56:37.857642 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:37.857643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:37.857655 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:37.857714 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:56:37.857835 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:37.858317 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:37.859039 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:37.859247 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:37.859302 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:37.859436 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:56:37.859461 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:56:37.859492 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:37.859532 (+ 40us) server_negotiation.cc:299] Negotiation successful
0619 08:56:37.859549 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":51,"threads_started":1}
I20260619 08:56:37.860402 8342 raft_consensus.cc:1261] T 7c5ebe0596ac4774b9494c2e4d3680d6 P eac8f4fc9d344cee8cf11be42b0c0b3b [term 1 FOLLOWER]: Refusing update from remote peer a770824c003e422f8fd8ef3d458332ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:56:37.860422 8050 raft_consensus.cc:1261] T 7c5ebe0596ac4774b9494c2e4d3680d6 P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 1 FOLLOWER]: Refusing update from remote peer a770824c003e422f8fd8ef3d458332ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:56:37.860607 8438 consensus_queue.cc:1035] T 7c5ebe0596ac4774b9494c2e4d3680d6 P a770824c003e422f8fd8ef3d458332ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:56:37.860750 8503 consensus_queue.cc:1035] T 7c5ebe0596ac4774b9494c2e4d3680d6 P a770824c003e422f8fd8ef3d458332ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:56:37.863029 7908 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:56:37.863116 7908 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:56:37.863765 7908 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57ff80e31aa34b1d880dacffdefa69ab: Sending DeleteTablet for 3 replicas of tablet 7c5ebe0596ac4774b9494c2e4d3680d6
I20260619 08:56:37.863984 8046 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c5ebe0596ac4774b9494c2e4d3680d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60324
I20260619 08:56:37.864014 8309 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c5ebe0596ac4774b9494c2e4d3680d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44430
I20260619 08:56:37.864100 8186 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c5ebe0596ac4774b9494c2e4d3680d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33474
I20260619 08:56:37.864187 8486 tablet_replica.cc:330] T 7c5ebe0596ac4774b9494c2e4d3680d6 P 33cbeb4cbd1840b5a05436a1aef6bf2c: stopping tablet replica
I20260619 08:56:37.864240 8486 raft_consensus.cc:2229] T 7c5ebe0596ac4774b9494c2e4d3680d6 P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:56:37.864284 8486 raft_consensus.cc:2258] T 7c5ebe0596ac4774b9494c2e4d3680d6 P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:37.864225 7908 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 08:56:37.864351 8485 tablet_replica.cc:330] T 7c5ebe0596ac4774b9494c2e4d3680d6 P a770824c003e422f8fd8ef3d458332ee: stopping tablet replica
W20260619 08:56:37.864364 7908 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:56:37.864396 8485 raft_consensus.cc:2229] T 7c5ebe0596ac4774b9494c2e4d3680d6 P a770824c003e422f8fd8ef3d458332ee [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:56:37.864445 8485 raft_consensus.cc:2258] T 7c5ebe0596ac4774b9494c2e4d3680d6 P a770824c003e422f8fd8ef3d458332ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:37.864454 8487 tablet_replica.cc:330] T 7c5ebe0596ac4774b9494c2e4d3680d6 P eac8f4fc9d344cee8cf11be42b0c0b3b: stopping tablet replica
I20260619 08:56:37.864503 8487 raft_consensus.cc:2229] T 7c5ebe0596ac4774b9494c2e4d3680d6 P eac8f4fc9d344cee8cf11be42b0c0b3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:56:37.864545 8487 raft_consensus.cc:2258] T 7c5ebe0596ac4774b9494c2e4d3680d6 P eac8f4fc9d344cee8cf11be42b0c0b3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:37.864761 8485 ts_tablet_manager.cc:1905] T 7c5ebe0596ac4774b9494c2e4d3680d6 P a770824c003e422f8fd8ef3d458332ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:37.864826 8487 ts_tablet_manager.cc:1905] T 7c5ebe0596ac4774b9494c2e4d3680d6 P eac8f4fc9d344cee8cf11be42b0c0b3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:37.865543 8186 tablet_service.cc:1467] Processing CreateTablet for tablet f45b9a11a1514eafb516389e17a7c1d3 (DEFAULT_TABLE table=test-table [id=e03d79143b314edcbd77b7264696fdf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:37.865543 8046 tablet_service.cc:1467] Processing CreateTablet for tablet f45b9a11a1514eafb516389e17a7c1d3 (DEFAULT_TABLE table=test-table [id=e03d79143b314edcbd77b7264696fdf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:37.865651 8046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f45b9a11a1514eafb516389e17a7c1d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:37.865651 8186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f45b9a11a1514eafb516389e17a7c1d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:37.866334 8487 ts_tablet_manager.cc:1918] T 7c5ebe0596ac4774b9494c2e4d3680d6 P eac8f4fc9d344cee8cf11be42b0c0b3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:56:37.866350 8486 ts_tablet_manager.cc:1905] T 7c5ebe0596ac4774b9494c2e4d3680d6 P 33cbeb4cbd1840b5a05436a1aef6bf2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:37.866394 8487 log.cc:1196] T 7c5ebe0596ac4774b9494c2e4d3680d6 P eac8f4fc9d344cee8cf11be42b0c0b3b: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-2/wal/wals/7c5ebe0596ac4774b9494c2e4d3680d6
I20260619 08:56:37.866572 8487 ts_tablet_manager.cc:1939] T 7c5ebe0596ac4774b9494c2e4d3680d6 P eac8f4fc9d344cee8cf11be42b0c0b3b: Deleting consensus metadata
I20260619 08:56:37.866931 8423 tablet_bootstrap.cc:492] T f45b9a11a1514eafb516389e17a7c1d3 P 33cbeb4cbd1840b5a05436a1aef6bf2c: Bootstrap starting.
I20260619 08:56:37.867075 8309 tablet_service.cc:1467] Processing CreateTablet for tablet f45b9a11a1514eafb516389e17a7c1d3 (DEFAULT_TABLE table=test-table [id=e03d79143b314edcbd77b7264696fdf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:37.867091 7895 catalog_manager.cc:4841] TS eac8f4fc9d344cee8cf11be42b0c0b3b (127.0.98.3:40723): tablet 7c5ebe0596ac4774b9494c2e4d3680d6 (table test-table [id=e48aa9db5a494599bf976ff82b93a97d]) successfully deleted
I20260619 08:56:37.867159 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f45b9a11a1514eafb516389e17a7c1d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:37.867249 8423 tablet_bootstrap.cc:654] T f45b9a11a1514eafb516389e17a7c1d3 P 33cbeb4cbd1840b5a05436a1aef6bf2c: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:37.867657 8486 ts_tablet_manager.cc:1918] T 7c5ebe0596ac4774b9494c2e4d3680d6 P 33cbeb4cbd1840b5a05436a1aef6bf2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:56:37.867722 8423 tablet_bootstrap.cc:492] T f45b9a11a1514eafb516389e17a7c1d3 P 33cbeb4cbd1840b5a05436a1aef6bf2c: No bootstrap required, opened a new log
I20260619 08:56:37.867781 8423 ts_tablet_manager.cc:1397] T f45b9a11a1514eafb516389e17a7c1d3 P 33cbeb4cbd1840b5a05436a1aef6bf2c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260619 08:56:37.867799 8485 ts_tablet_manager.cc:1918] T 7c5ebe0596ac4774b9494c2e4d3680d6 P a770824c003e422f8fd8ef3d458332ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:56:37.867846 8485 log.cc:1196] T 7c5ebe0596ac4774b9494c2e4d3680d6 P a770824c003e422f8fd8ef3d458332ee: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-1/wal/wals/7c5ebe0596ac4774b9494c2e4d3680d6
I20260619 08:56:37.867889 8423 raft_consensus.cc:348] T f45b9a11a1514eafb516389e17a7c1d3 P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } }
I20260619 08:56:37.867959 8423 raft_consensus.cc:374] T f45b9a11a1514eafb516389e17a7c1d3 P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:37.867978 8423 raft_consensus.cc:729] T f45b9a11a1514eafb516389e17a7c1d3 P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33cbeb4cbd1840b5a05436a1aef6bf2c, State: Initialized, Role: FOLLOWER
I20260619 08:56:37.868033 8485 ts_tablet_manager.cc:1939] T 7c5ebe0596ac4774b9494c2e4d3680d6 P a770824c003e422f8fd8ef3d458332ee: Deleting consensus metadata
I20260619 08:56:37.868016 8423 consensus_queue.cc:260] T f45b9a11a1514eafb516389e17a7c1d3 P 33cbeb4cbd1840b5a05436a1aef6bf2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } }
I20260619 08:56:37.868124 8423 ts_tablet_manager.cc:1428] T f45b9a11a1514eafb516389e17a7c1d3 P 33cbeb4cbd1840b5a05436a1aef6bf2c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:37.868315 7893 catalog_manager.cc:4841] TS a770824c003e422f8fd8ef3d458332ee (127.0.98.2:36709): tablet 7c5ebe0596ac4774b9494c2e4d3680d6 (table test-table [id=e48aa9db5a494599bf976ff82b93a97d]) successfully deleted
I20260619 08:56:37.868335 8420 tablet_bootstrap.cc:492] T f45b9a11a1514eafb516389e17a7c1d3 P eac8f4fc9d344cee8cf11be42b0c0b3b: Bootstrap starting.
I20260619 08:56:37.868695 8421 tablet_bootstrap.cc:492] T f45b9a11a1514eafb516389e17a7c1d3 P a770824c003e422f8fd8ef3d458332ee: Bootstrap starting.
I20260619 08:56:37.868891 8420 tablet_bootstrap.cc:654] T f45b9a11a1514eafb516389e17a7c1d3 P eac8f4fc9d344cee8cf11be42b0c0b3b: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:37.868978 8421 tablet_bootstrap.cc:654] T f45b9a11a1514eafb516389e17a7c1d3 P a770824c003e422f8fd8ef3d458332ee: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:37.867720 8486 log.cc:1196] T 7c5ebe0596ac4774b9494c2e4d3680d6 P 33cbeb4cbd1840b5a05436a1aef6bf2c: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-0/wal/wals/7c5ebe0596ac4774b9494c2e4d3680d6
I20260619 08:56:37.869486 8421 tablet_bootstrap.cc:492] T f45b9a11a1514eafb516389e17a7c1d3 P a770824c003e422f8fd8ef3d458332ee: No bootstrap required, opened a new log
I20260619 08:56:37.869522 8421 ts_tablet_manager.cc:1397] T f45b9a11a1514eafb516389e17a7c1d3 P a770824c003e422f8fd8ef3d458332ee: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 08:56:37.869619 8486 ts_tablet_manager.cc:1939] T 7c5ebe0596ac4774b9494c2e4d3680d6 P 33cbeb4cbd1840b5a05436a1aef6bf2c: Deleting consensus metadata
I20260619 08:56:37.869613 8421 raft_consensus.cc:348] T f45b9a11a1514eafb516389e17a7c1d3 P a770824c003e422f8fd8ef3d458332ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } }
I20260619 08:56:37.869673 8421 raft_consensus.cc:374] T f45b9a11a1514eafb516389e17a7c1d3 P a770824c003e422f8fd8ef3d458332ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:37.869690 8421 raft_consensus.cc:729] T f45b9a11a1514eafb516389e17a7c1d3 P a770824c003e422f8fd8ef3d458332ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a770824c003e422f8fd8ef3d458332ee, State: Initialized, Role: FOLLOWER
I20260619 08:56:37.869733 8421 consensus_queue.cc:260] T f45b9a11a1514eafb516389e17a7c1d3 P a770824c003e422f8fd8ef3d458332ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } }
I20260619 08:56:37.869818 8421 ts_tablet_manager.cc:1428] T f45b9a11a1514eafb516389e17a7c1d3 P a770824c003e422f8fd8ef3d458332ee: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:37.869897 7895 catalog_manager.cc:4841] TS 33cbeb4cbd1840b5a05436a1aef6bf2c (127.0.98.1:45395): tablet 7c5ebe0596ac4774b9494c2e4d3680d6 (table test-table [id=e48aa9db5a494599bf976ff82b93a97d]) successfully deleted
I20260619 08:56:37.870024 8420 tablet_bootstrap.cc:492] T f45b9a11a1514eafb516389e17a7c1d3 P eac8f4fc9d344cee8cf11be42b0c0b3b: No bootstrap required, opened a new log
I20260619 08:56:37.870072 8420 ts_tablet_manager.cc:1397] T f45b9a11a1514eafb516389e17a7c1d3 P eac8f4fc9d344cee8cf11be42b0c0b3b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 08:56:37.870218 8420 raft_consensus.cc:348] T f45b9a11a1514eafb516389e17a7c1d3 P eac8f4fc9d344cee8cf11be42b0c0b3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } }
I20260619 08:56:37.870265 8420 raft_consensus.cc:374] T f45b9a11a1514eafb516389e17a7c1d3 P eac8f4fc9d344cee8cf11be42b0c0b3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:37.870276 8420 raft_consensus.cc:729] T f45b9a11a1514eafb516389e17a7c1d3 P eac8f4fc9d344cee8cf11be42b0c0b3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eac8f4fc9d344cee8cf11be42b0c0b3b, State: Initialized, Role: FOLLOWER
I20260619 08:56:37.870329 8420 consensus_queue.cc:260] T f45b9a11a1514eafb516389e17a7c1d3 P eac8f4fc9d344cee8cf11be42b0c0b3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } }
I20260619 08:56:37.870412 8420 ts_tablet_manager.cc:1428] T f45b9a11a1514eafb516389e17a7c1d3 P eac8f4fc9d344cee8cf11be42b0c0b3b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:38.116186 8438 raft_consensus.cc:482] T f45b9a11a1514eafb516389e17a7c1d3 P a770824c003e422f8fd8ef3d458332ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:56:38.116291 8438 raft_consensus.cc:504] T f45b9a11a1514eafb516389e17a7c1d3 P a770824c003e422f8fd8ef3d458332ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } }
I20260619 08:56:38.116449 8438 leader_election.cc:288] T f45b9a11a1514eafb516389e17a7c1d3 P a770824c003e422f8fd8ef3d458332ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33cbeb4cbd1840b5a05436a1aef6bf2c (127.0.98.1:45395), eac8f4fc9d344cee8cf11be42b0c0b3b (127.0.98.3:40723)
I20260619 08:56:38.116663 8050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f45b9a11a1514eafb516389e17a7c1d3" candidate_uuid: "a770824c003e422f8fd8ef3d458332ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" is_pre_election: true
I20260619 08:56:38.116710 8346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f45b9a11a1514eafb516389e17a7c1d3" candidate_uuid: "a770824c003e422f8fd8ef3d458332ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" is_pre_election: true
I20260619 08:56:38.116770 8050 raft_consensus.cc:2454] T f45b9a11a1514eafb516389e17a7c1d3 P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a770824c003e422f8fd8ef3d458332ee in term 0.
I20260619 08:56:38.116791 8346 raft_consensus.cc:2454] T f45b9a11a1514eafb516389e17a7c1d3 P eac8f4fc9d344cee8cf11be42b0c0b3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a770824c003e422f8fd8ef3d458332ee in term 0.
I20260619 08:56:38.116919 8139 leader_election.cc:302] T f45b9a11a1514eafb516389e17a7c1d3 P a770824c003e422f8fd8ef3d458332ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33cbeb4cbd1840b5a05436a1aef6bf2c, a770824c003e422f8fd8ef3d458332ee; no voters:
I20260619 08:56:38.117024 8438 raft_consensus.cc:2781] T f45b9a11a1514eafb516389e17a7c1d3 P a770824c003e422f8fd8ef3d458332ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:56:38.117089 8438 raft_consensus.cc:482] T f45b9a11a1514eafb516389e17a7c1d3 P a770824c003e422f8fd8ef3d458332ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:56:38.117106 8438 raft_consensus.cc:3037] T f45b9a11a1514eafb516389e17a7c1d3 P a770824c003e422f8fd8ef3d458332ee [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:38.117729 8438 raft_consensus.cc:504] T f45b9a11a1514eafb516389e17a7c1d3 P a770824c003e422f8fd8ef3d458332ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } }
I20260619 08:56:38.117841 8438 leader_election.cc:288] T f45b9a11a1514eafb516389e17a7c1d3 P a770824c003e422f8fd8ef3d458332ee [CANDIDATE]: Term 1 election: Requested vote from peers 33cbeb4cbd1840b5a05436a1aef6bf2c (127.0.98.1:45395), eac8f4fc9d344cee8cf11be42b0c0b3b (127.0.98.3:40723)
I20260619 08:56:38.117988 8050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f45b9a11a1514eafb516389e17a7c1d3" candidate_uuid: "a770824c003e422f8fd8ef3d458332ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c"
I20260619 08:56:38.118003 8346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f45b9a11a1514eafb516389e17a7c1d3" candidate_uuid: "a770824c003e422f8fd8ef3d458332ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b"
I20260619 08:56:38.118064 8346 raft_consensus.cc:3037] T f45b9a11a1514eafb516389e17a7c1d3 P eac8f4fc9d344cee8cf11be42b0c0b3b [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:38.118064 8050 raft_consensus.cc:3037] T f45b9a11a1514eafb516389e17a7c1d3 P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:38.118834 8346 raft_consensus.cc:2454] T f45b9a11a1514eafb516389e17a7c1d3 P eac8f4fc9d344cee8cf11be42b0c0b3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a770824c003e422f8fd8ef3d458332ee in term 1.
I20260619 08:56:38.118834 8050 raft_consensus.cc:2454] T f45b9a11a1514eafb516389e17a7c1d3 P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a770824c003e422f8fd8ef3d458332ee in term 1.
I20260619 08:56:38.118984 8139 leader_election.cc:302] T f45b9a11a1514eafb516389e17a7c1d3 P a770824c003e422f8fd8ef3d458332ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a770824c003e422f8fd8ef3d458332ee, eac8f4fc9d344cee8cf11be42b0c0b3b; no voters:
I20260619 08:56:38.119077 8438 raft_consensus.cc:2781] T f45b9a11a1514eafb516389e17a7c1d3 P a770824c003e422f8fd8ef3d458332ee [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:38.119130 8438 raft_consensus.cc:686] T f45b9a11a1514eafb516389e17a7c1d3 P a770824c003e422f8fd8ef3d458332ee [term 1 LEADER]: Becoming Leader. State: Replica: a770824c003e422f8fd8ef3d458332ee, State: Running, Role: LEADER
I20260619 08:56:38.119174 8438 consensus_queue.cc:237] T f45b9a11a1514eafb516389e17a7c1d3 P a770824c003e422f8fd8ef3d458332ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } }
I20260619 08:56:38.119735 7908 catalog_manager.cc:5495] T f45b9a11a1514eafb516389e17a7c1d3 P a770824c003e422f8fd8ef3d458332ee reported cstate change: term changed from 0 to 1, leader changed from <none> to a770824c003e422f8fd8ef3d458332ee (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a770824c003e422f8fd8ef3d458332ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a770824c003e422f8fd8ef3d458332ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 } health_report { overall_health: UNKNOWN } } }
I20260619 08:56:38.154675 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:38.152282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45395 (local address 127.0.98.2:55159)
0619 08:56:38.152437 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:38.152449 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:38.152511 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:38.152631 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:38.152631 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:38.152633 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 08:56:38.152686 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:38.152687 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:38.153323 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:38.153324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:38.153943 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:38.153945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:38.154281 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:38.154282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:38.154346 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:38.154359 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 08:56:38.154515 (+ 156us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 08:56:38.154517 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:56:38.154553 (+ 36us) client_negotiation.cc:240] Negotiation successful
0619 08:56:38.154603 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":77,"threads_started":1}
I20260619 08:56:38.154671 8519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:38.152341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55159 (local address 127.0.98.1:45395)
0619 08:56:38.152459 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:38.152461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:38.152498 (+ 37us) server_negotiation.cc:407] Connection header received
0619 08:56:38.152560 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:38.152561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:38.152572 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:38.152630 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:56:38.152780 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:38.153252 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:38.154022 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:38.154215 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:38.154272 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:38.154433 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:56:38.154465 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:56:38.154504 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:38.154566 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 08:56:38.154587 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":51,"threads_started":1}
I20260619 08:56:38.156482 8050 raft_consensus.cc:1261] T f45b9a11a1514eafb516389e17a7c1d3 P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 1 FOLLOWER]: Refusing update from remote peer a770824c003e422f8fd8ef3d458332ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:56:38.156519 8346 raft_consensus.cc:1261] T f45b9a11a1514eafb516389e17a7c1d3 P eac8f4fc9d344cee8cf11be42b0c0b3b [term 1 FOLLOWER]: Refusing update from remote peer a770824c003e422f8fd8ef3d458332ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:56:38.156680 8438 consensus_queue.cc:1035] T f45b9a11a1514eafb516389e17a7c1d3 P a770824c003e422f8fd8ef3d458332ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "33cbeb4cbd1840b5a05436a1aef6bf2c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:56:38.156795 8509 consensus_queue.cc:1035] T f45b9a11a1514eafb516389e17a7c1d3 P a770824c003e422f8fd8ef3d458332ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "eac8f4fc9d344cee8cf11be42b0c0b3b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:56:38.163002 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:38.160770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45605 (local address 127.0.98.2:36709)
0619 08:56:38.160809 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:38.160811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:38.160934 (+ 123us) server_negotiation.cc:407] Connection header received
0619 08:56:38.161031 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:38.161032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:38.161042 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:38.161081 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:56:38.161165 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:38.161606 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:38.162286 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:38.162506 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:38.162568 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:38.162743 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:56:38.162778 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:56:38.162824 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:38.162910 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 08:56:38.162933 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 08:56:38.163004 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:38.160737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36709 (local address 127.0.98.1:45605)
0619 08:56:38.160878 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:38.160890 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:38.160946 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:38.161086 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:38.161087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:38.161089 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 08:56:38.161127 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:38.161128 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:38.161652 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:38.161652 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:38.162202 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:38.162204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:38.162627 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:38.162628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:38.162675 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:38.162679 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 08:56:38.162834 (+ 155us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 08:56:38.162836 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:56:38.162874 (+ 38us) client_negotiation.cc:240] Negotiation successful
0619 08:56:38.162922 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
I20260619 08:56:38.168418 7908 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:56:38.168514 7908 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:56:38.169031 7908 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57ff80e31aa34b1d880dacffdefa69ab: Sending DeleteTablet for 3 replicas of tablet f45b9a11a1514eafb516389e17a7c1d3
I20260619 08:56:38.169240 8046 tablet_service.cc:1514] Processing DeleteTablet for tablet f45b9a11a1514eafb516389e17a7c1d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:38 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60324
I20260619 08:56:38.169277 8186 tablet_service.cc:1514] Processing DeleteTablet for tablet f45b9a11a1514eafb516389e17a7c1d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:38 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33474
I20260619 08:56:38.169310 8309 tablet_service.cc:1514] Processing DeleteTablet for tablet f45b9a11a1514eafb516389e17a7c1d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:38 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44430
I20260619 08:56:38.169335 8486 tablet_replica.cc:330] T f45b9a11a1514eafb516389e17a7c1d3 P 33cbeb4cbd1840b5a05436a1aef6bf2c: stopping tablet replica
I20260619 08:56:38.169366 8485 tablet_replica.cc:330] T f45b9a11a1514eafb516389e17a7c1d3 P a770824c003e422f8fd8ef3d458332ee: stopping tablet replica
I20260619 08:56:38.169381 8486 raft_consensus.cc:2229] T f45b9a11a1514eafb516389e17a7c1d3 P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:56:38.169384 8487 tablet_replica.cc:330] T f45b9a11a1514eafb516389e17a7c1d3 P eac8f4fc9d344cee8cf11be42b0c0b3b: stopping tablet replica
I20260619 08:56:38.169409 8485 raft_consensus.cc:2229] T f45b9a11a1514eafb516389e17a7c1d3 P a770824c003e422f8fd8ef3d458332ee [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:56:38.169413 8486 raft_consensus.cc:2258] T f45b9a11a1514eafb516389e17a7c1d3 P 33cbeb4cbd1840b5a05436a1aef6bf2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:38.169427 8487 raft_consensus.cc:2229] T f45b9a11a1514eafb516389e17a7c1d3 P eac8f4fc9d344cee8cf11be42b0c0b3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:56:38.169456 8485 raft_consensus.cc:2258] T f45b9a11a1514eafb516389e17a7c1d3 P a770824c003e422f8fd8ef3d458332ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:38.169456 8487 raft_consensus.cc:2258] T f45b9a11a1514eafb516389e17a7c1d3 P eac8f4fc9d344cee8cf11be42b0c0b3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:38.169691 8485 ts_tablet_manager.cc:1905] T f45b9a11a1514eafb516389e17a7c1d3 P a770824c003e422f8fd8ef3d458332ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:38.169800 8487 ts_tablet_manager.cc:1905] T f45b9a11a1514eafb516389e17a7c1d3 P eac8f4fc9d344cee8cf11be42b0c0b3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:38.170861 8486 ts_tablet_manager.cc:1905] T f45b9a11a1514eafb516389e17a7c1d3 P 33cbeb4cbd1840b5a05436a1aef6bf2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:38.171134 8485 ts_tablet_manager.cc:1918] T f45b9a11a1514eafb516389e17a7c1d3 P a770824c003e422f8fd8ef3d458332ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 08:56:38.171159 8487 ts_tablet_manager.cc:1918] T f45b9a11a1514eafb516389e17a7c1d3 P eac8f4fc9d344cee8cf11be42b0c0b3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 08:56:38.171183 8485 log.cc:1196] T f45b9a11a1514eafb516389e17a7c1d3 P a770824c003e422f8fd8ef3d458332ee: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-1/wal/wals/f45b9a11a1514eafb516389e17a7c1d3
I20260619 08:56:38.171202 8487 log.cc:1196] T f45b9a11a1514eafb516389e17a7c1d3 P eac8f4fc9d344cee8cf11be42b0c0b3b: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-2/wal/wals/f45b9a11a1514eafb516389e17a7c1d3
I20260619 08:56:38.171387 8485 ts_tablet_manager.cc:1939] T f45b9a11a1514eafb516389e17a7c1d3 P a770824c003e422f8fd8ef3d458332ee: Deleting consensus metadata
I20260619 08:56:38.171407 8487 ts_tablet_manager.cc:1939] T f45b9a11a1514eafb516389e17a7c1d3 P eac8f4fc9d344cee8cf11be42b0c0b3b: Deleting consensus metadata
I20260619 08:56:38.171628 7893 catalog_manager.cc:4841] TS a770824c003e422f8fd8ef3d458332ee (127.0.98.2:36709): tablet f45b9a11a1514eafb516389e17a7c1d3 (table test-table [id=e03d79143b314edcbd77b7264696fdf6]) successfully deleted
I20260619 08:56:38.171808 7895 catalog_manager.cc:4841] TS eac8f4fc9d344cee8cf11be42b0c0b3b (127.0.98.3:40723): tablet f45b9a11a1514eafb516389e17a7c1d3 (table test-table [id=e03d79143b314edcbd77b7264696fdf6]) successfully deleted
I20260619 08:56:38.172184 8486 ts_tablet_manager.cc:1918] T f45b9a11a1514eafb516389e17a7c1d3 P 33cbeb4cbd1840b5a05436a1aef6bf2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 08:56:38.172230 8486 log.cc:1196] T f45b9a11a1514eafb516389e17a7c1d3 P 33cbeb4cbd1840b5a05436a1aef6bf2c: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859361540516-392-0/minicluster-data/ts-0/wal/wals/f45b9a11a1514eafb516389e17a7c1d3
I20260619 08:56:38.172416 8486 ts_tablet_manager.cc:1939] T f45b9a11a1514eafb516389e17a7c1d3 P 33cbeb4cbd1840b5a05436a1aef6bf2c: Deleting consensus metadata
I20260619 08:56:38.172559 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 7981
W20260619 08:56:38.178104 7895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45395 (error 108)
I20260619 08:56:38.178311 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 8121
I20260619 08:56:38.180054 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:38.178413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45395 (local address 127.0.0.1:60366)
0619 08:56:38.179963 (+ 1550us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:38.180009 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1532,"thread_start_us":38,"threads_started":1}
W20260619 08:56:38.180191 7895 catalog_manager.cc:4584] TS 33cbeb4cbd1840b5a05436a1aef6bf2c (127.0.98.1:45395): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f45b9a11a1514eafb516389e17a7c1d3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45395: connect: Connection refused (error 111)
I20260619 08:56:38.183276 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 8261
I20260619 08:56:38.187461 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 7877
2026-06-19T08:56:38Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3265 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:56:38 dist-test-slave-h4g5 krb5kdc[8529](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:56:38 dist-test-slave-h4g5 krb5kdc[8529](info): set up 2 sockets
Jun 19 08:56:38 dist-test-slave-h4g5 krb5kdc[8529](info): commencing operation
krb5kdc: starting...
W20260619 08:56:39.136324 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 08:56:39 dist-test-slave-h4g5 krb5kdc[8529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:56:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:56:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 08:56:39.246729 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44815
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:39.327243 8542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:39.327437 8542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:39.327457 8542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:39.328648 8542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:56:39.328689 8542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:39.328702 8542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:39.328715 8542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:56:39.328738 8542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:56:39.330225 8542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:39.330448 8542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:39.330675 8542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 08:56:39.335286 8542 server_base.cc:1034] running on GCE node
W20260619 08:56:39.335240 8550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:39.335228 8547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:39.335266 8548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:39.335709 8542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:39.335964 8542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:39.337091 8542 hybrid_clock.cc:648] HybridClock initialized: now 1781859399337080 us; error 26 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T08:56:39Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1157 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:56:39 dist-test-slave-h4g5 krb5kdc[8555](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:56:39 dist-test-slave-h4g5 krb5kdc[8555](info): set up 2 sockets
Jun 19 08:56:39 dist-test-slave-h4g5 krb5kdc[8555](info): commencing operation
krb5kdc: starting...
W20260619 08:56:40.308350 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 08:56:40 dist-test-slave-h4g5 krb5kdc[8555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:56:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:56:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 08:56:40.418344 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41709
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41709
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:40.496292 8568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:40.496495 8568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:40.496516 8568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:40.497720 8568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:56:40.497766 8568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:40.497777 8568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:40.497788 8568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:56:40.497799 8568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:56:40.499282 8568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:40.499502 8568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:40.499715 8568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:40.504287 8574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:40.504395 8576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:40.504302 8568 server_base.cc:1034] running on GCE node
W20260619 08:56:40.504323 8573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:40.504767 8568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:40.504997 8568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:40.506135 8568 hybrid_clock.cc:648] HybridClock initialized: now 1781859400506123 us; error 27 us; skew 500 ppm
Jun 19 08:56:40 dist-test-slave-h4g5 krb5kdc[8555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:40.508383 8568 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 08:56:40.508810 8568 webserver.cc:460] Webserver started at http://127.0.98.62:33629/ using document root <none> and password file <none>
I20260619 08:56:40.509016 8568 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:40.509061 8568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:40.509161 8568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:40.510187 8568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "e938aee6d2c746f2aa9ee9b88ea2167e"
format_stamp: "Formatted at 2026-06-19 08:56:40 on dist-test-slave-h4g5"
server_key: "00e009499b20c42e1b1c7566d50a8c75"
server_key_iv: "2a3261b3853231755d2a13806a742855"
server_key_version: "encryptionkey@0"
I20260619 08:56:40.510500 8568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "e938aee6d2c746f2aa9ee9b88ea2167e"
format_stamp: "Formatted at 2026-06-19 08:56:40 on dist-test-slave-h4g5"
server_key: "00e009499b20c42e1b1c7566d50a8c75"
server_key_iv: "2a3261b3853231755d2a13806a742855"
server_key_version: "encryptionkey@0"
I20260619 08:56:40.511683 8568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 08:56:40.512324 8583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:40.512442 8568 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:40.512517 8568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "e938aee6d2c746f2aa9ee9b88ea2167e"
format_stamp: "Formatted at 2026-06-19 08:56:40 on dist-test-slave-h4g5"
server_key: "00e009499b20c42e1b1c7566d50a8c75"
server_key_iv: "2a3261b3853231755d2a13806a742855"
server_key_version: "encryptionkey@0"
I20260619 08:56:40.512599 8568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:40.522356 8568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:40.524049 8568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:40.524207 8568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:40.527954 8568 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41709
I20260619 08:56:40.527994 8635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41709 every 8 connection(s)
I20260619 08:56:40.528503 8568 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:56:40.529170 8636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:40.531358 8636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e: Bootstrap starting.
I20260619 08:56:40.531879 8636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:40.532151 8636 log.cc:824] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:40.532507 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 8568
I20260619 08:56:40.532618 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 08:56:40.532789 392 external_mini_cluster.cc:1428] Setting key 2aca2363b10aee0431365f4cff20a65f
I20260619 08:56:40.532829 8636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e: No bootstrap required, opened a new log
I20260619 08:56:40.535409 8636 raft_consensus.cc:348] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e938aee6d2c746f2aa9ee9b88ea2167e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41709 } }
I20260619 08:56:40.535548 8636 raft_consensus.cc:374] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:40.535579 8636 raft_consensus.cc:729] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e938aee6d2c746f2aa9ee9b88ea2167e, State: Initialized, Role: FOLLOWER
I20260619 08:56:40.535697 8636 consensus_queue.cc:260] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e938aee6d2c746f2aa9ee9b88ea2167e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41709 } }
I20260619 08:56:40.535768 8636 raft_consensus.cc:388] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:56:40.535799 8636 raft_consensus.cc:482] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:56:40.535844 8636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:40.536587 8636 raft_consensus.cc:504] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e938aee6d2c746f2aa9ee9b88ea2167e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41709 } }
I20260619 08:56:40.536725 8636 leader_election.cc:302] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e938aee6d2c746f2aa9ee9b88ea2167e; no voters:
I20260619 08:56:40.536880 8636 leader_election.cc:288] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:56:40.536914 8641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:40.537082 8641 raft_consensus.cc:686] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e [term 1 LEADER]: Becoming Leader. State: Replica: e938aee6d2c746f2aa9ee9b88ea2167e, State: Running, Role: LEADER
I20260619 08:56:40.537158 8636 sys_catalog.cc:564] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:56:40.537189 8641 consensus_queue.cc:237] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e938aee6d2c746f2aa9ee9b88ea2167e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41709 } }
I20260619 08:56:40.537504 8641 sys_catalog.cc:455] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e938aee6d2c746f2aa9ee9b88ea2167e. Latest consensus state: current_term: 1 leader_uuid: "e938aee6d2c746f2aa9ee9b88ea2167e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e938aee6d2c746f2aa9ee9b88ea2167e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41709 } } }
I20260619 08:56:40.537551 8642 sys_catalog.cc:455] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e938aee6d2c746f2aa9ee9b88ea2167e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e938aee6d2c746f2aa9ee9b88ea2167e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41709 } } }
I20260619 08:56:40.537597 8642 sys_catalog.cc:458] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:40.537586 8641 sys_catalog.cc:458] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:40.537801 8645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:56:40.538316 8645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:56:40.539845 8645 catalog_manager.cc:1347] Generated new cluster ID: 5d6966e9d98a4fd79417c6004e7d9808
I20260619 08:56:40.539932 8645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 08:56:40 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:40.543609 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:40.533304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43252 (local address 127.0.98.62:41709)
0619 08:56:40.533458 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:40.533468 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:40.533493 (+ 25us) server_negotiation.cc:407] Connection header received
0619 08:56:40.534069 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:40.534076 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:40.534125 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:40.534238 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:40.534986 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:40.535972 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:40.537102 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:40.537742 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:40.537840 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:40.540734 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:40.540738 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:40.540738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:40.540750 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:40.542257 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:40.542562 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:40.542565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:40.542565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:40.542648 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:40.542813 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:40.542813 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:40.542814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:40.543048 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:40.543186 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:40.543289 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 08:56:40.543416 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260619 08:56:40.591131 8645 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:56:40.591710 8645 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:56:40.598310 8645 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e: Generated new TSK 0
I20260619 08:56:40.598493 8645 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 08:56:40.626559 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41709
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:40.705549 8662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:40.705729 8662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:40.705749 8662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:40.706862 8662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:40.706902 8662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:40.706929 8662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 08:56:40.708410 8662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:40.708627 8662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:40.708818 8662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:40.713413 8667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:40.713454 8668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:40.713641 8662 server_base.cc:1034] running on GCE node
W20260619 08:56:40.713810 8670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:40.714058 8662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:40.714278 8662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:40.715406 8662 hybrid_clock.cc:648] HybridClock initialized: now 1781859400715394 us; error 29 us; skew 500 ppm
Jun 19 08:56:40 dist-test-slave-h4g5 krb5kdc[8555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:40.717720 8662 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 08:56:40.718099 8662 webserver.cc:460] Webserver started at http://127.0.98.1:33867/ using document root <none> and password file <none>
I20260619 08:56:40.718283 8662 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:40.718328 8662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:40.718443 8662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:40.719395 8662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-0/data/instance:
uuid: "d3d183f5688940829d692835e9f4b54e"
format_stamp: "Formatted at 2026-06-19 08:56:40 on dist-test-slave-h4g5"
server_key: "2ef1e883ed00538f3047bfed8e537364"
server_key_iv: "d04b4e97a5668f13e4356fe2e8de5400"
server_key_version: "encryptionkey@0"
I20260619 08:56:40.719707 8662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d3d183f5688940829d692835e9f4b54e"
format_stamp: "Formatted at 2026-06-19 08:56:40 on dist-test-slave-h4g5"
server_key: "2ef1e883ed00538f3047bfed8e537364"
server_key_iv: "d04b4e97a5668f13e4356fe2e8de5400"
server_key_version: "encryptionkey@0"
I20260619 08:56:40.720906 8662 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:56:40.721549 8677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:40.721719 8662 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 08:56:40.721809 8662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-0/wal
uuid: "d3d183f5688940829d692835e9f4b54e"
format_stamp: "Formatted at 2026-06-19 08:56:40 on dist-test-slave-h4g5"
server_key: "2ef1e883ed00538f3047bfed8e537364"
server_key_iv: "d04b4e97a5668f13e4356fe2e8de5400"
server_key_version: "encryptionkey@0"
I20260619 08:56:40.721869 8662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:40.730168 8662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:40.732177 8662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:40.732306 8662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:40.732522 8662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:40.732788 8662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:40.732820 8662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:40.732846 8662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:40.732864 8662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:40.738260 8662 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43473
I20260619 08:56:40.738327 8790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43473 every 8 connection(s)
I20260619 08:56:40.738611 8662 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
I20260619 08:56:40.740734 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 8662
I20260619 08:56:40.740821 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-0/wal/instance
I20260619 08:56:40.740975 392 external_mini_cluster.cc:1428] Setting key 04dbc2a9c72a79a51a6d95c7a479594e
Jun 19 08:56:40 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:40.748005 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:40.739313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51265 (local address 127.0.98.62:41709)
0619 08:56:40.739376 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:40.739378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:40.739565 (+ 187us) server_negotiation.cc:407] Connection header received
0619 08:56:40.740531 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:40.740532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:40.740545 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:40.740597 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:40.741265 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:40.741771 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:40.742884 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:40.743091 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:40.743145 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:40.745494 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:40.745495 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:40.745495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:40.745502 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:40.746702 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:40.747147 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:40.747147 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:40.747148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:40.747193 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:40.747436 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:40.747436 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:40.747436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:40.747526 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:40.747581 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:40.747873 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 08:56:40.747940 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 08:56:40.748324 8793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:40.739282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.1:51265)
0619 08:56:40.739450 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:40.739485 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:40.740419 (+ 934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:40.740665 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:40.740668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:40.740955 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:40.741178 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:40.741183 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:40.741853 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:40.741854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:40.742804 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:40.742807 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:40.743161 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:40.743162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:40.743296 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:40.743793 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:40.743802 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:40.745406 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:40.746786 (+ 1380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:40.746793 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:40.746795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:40.747080 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:40.747248 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:40.747249 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:40.747249 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:40.747370 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:40.747581 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:40.747587 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:40.747747 (+ 160us) client_negotiation.cc:769] Sending connection context
0619 08:56:40.747867 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 08:56:40.748031 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 08:56:40.748795 8791 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41709
I20260619 08:56:40.748879 8791 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:40.749017 8791 heartbeater.cc:511] Master 127.0.98.62:41709 requested a full tablet report, sending...
I20260619 08:56:40.749444 8600 ts_manager.cc:194] Registered new tserver with Master: d3d183f5688940829d692835e9f4b54e (127.0.98.1:43473)
I20260619 08:56:40.749989 8600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51265
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 08:56:40.762939 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41709
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:40.840628 8796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:40.840826 8796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:40.840855 8796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:40.842005 8796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:40.842046 8796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:40.842080 8796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 08:56:40.843528 8796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:40.843753 8796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:40.843956 8796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:40.848671 8804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:40.848671 8802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:40.848675 8801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:40.848762 8796 server_base.cc:1034] running on GCE node
I20260619 08:56:40.849138 8796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:40.849351 8796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:40.850522 8796 hybrid_clock.cc:648] HybridClock initialized: now 1781859400850505 us; error 25 us; skew 500 ppm
Jun 19 08:56:40 dist-test-slave-h4g5 krb5kdc[8555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:40.853003 8796 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 08:56:40.853418 8796 webserver.cc:460] Webserver started at http://127.0.98.2:46547/ using document root <none> and password file <none>
I20260619 08:56:40.853605 8796 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:40.853646 8796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:40.853731 8796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:40.854705 8796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-1/data/instance:
uuid: "700bc936a7384bb89f48a2713b9ad79a"
format_stamp: "Formatted at 2026-06-19 08:56:40 on dist-test-slave-h4g5"
server_key: "0038410e28547783394bfa0eba7d46c4"
server_key_iv: "1a370c23e888f271a260ab40bc556b6f"
server_key_version: "encryptionkey@0"
I20260619 08:56:40.854982 8796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-1/wal/instance:
uuid: "700bc936a7384bb89f48a2713b9ad79a"
format_stamp: "Formatted at 2026-06-19 08:56:40 on dist-test-slave-h4g5"
server_key: "0038410e28547783394bfa0eba7d46c4"
server_key_iv: "1a370c23e888f271a260ab40bc556b6f"
server_key_version: "encryptionkey@0"
I20260619 08:56:40.856174 8796 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:56:40.856887 8811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:40.857071 8796 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:40.857153 8796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-1/wal
uuid: "700bc936a7384bb89f48a2713b9ad79a"
format_stamp: "Formatted at 2026-06-19 08:56:40 on dist-test-slave-h4g5"
server_key: "0038410e28547783394bfa0eba7d46c4"
server_key_iv: "1a370c23e888f271a260ab40bc556b6f"
server_key_version: "encryptionkey@0"
I20260619 08:56:40.857208 8796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:40.871609 8796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:40.873369 8796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:40.873485 8796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:40.873660 8796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:40.873917 8796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:40.873947 8796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:40.873967 8796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:40.873986 8796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:40.879103 8796 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43679
I20260619 08:56:40.879181 8924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43679 every 8 connection(s)
I20260619 08:56:40.879447 8796 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 08:56:40 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:40.887142 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 8796
I20260619 08:56:40.887226 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-1/wal/instance
I20260619 08:56:40.887394 392 external_mini_cluster.cc:1428] Setting key 2a126b24027e5da91361d02490576cee
I20260619 08:56:40.888257 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:40.880132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53519 (local address 127.0.98.62:41709)
0619 08:56:40.880180 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:40.880182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:40.880392 (+ 210us) server_negotiation.cc:407] Connection header received
0619 08:56:40.881343 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:40.881345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:40.881358 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:40.881409 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:40.882105 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:40.882608 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:40.883584 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:40.883767 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:40.883821 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:40.885826 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:40.885828 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:40.885828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:40.885835 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:40.886925 (+ 1090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:40.887284 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:40.887285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:40.887286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:40.887338 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:40.887579 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:40.887580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:40.887581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:40.887699 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:40.887772 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:40.888053 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 08:56:40.888166 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 08:56:40.888458 8927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:40.880125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.2:53519)
0619 08:56:40.880275 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:40.880309 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:40.881236 (+ 927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:40.881491 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:40.881495 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:40.881792 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:40.881995 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:40.881999 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:40.882695 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:40.882696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:40.883497 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:40.883499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:40.883833 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:40.883834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:40.883960 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:40.884428 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:40.884436 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:40.885738 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:40.886992 (+ 1254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:40.886999 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:40.887001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:40.887217 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:40.887390 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:40.887390 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:40.887391 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:40.887520 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:40.887762 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:40.887769 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:40.887918 (+ 149us) client_negotiation.cc:769] Sending connection context
0619 08:56:40.888028 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 08:56:40.888214 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":67,"threads_started":1}
I20260619 08:56:40.888909 8925 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41709
I20260619 08:56:40.888986 8925 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:40.889147 8925 heartbeater.cc:511] Master 127.0.98.62:41709 requested a full tablet report, sending...
I20260619 08:56:40.889489 8600 ts_manager.cc:194] Registered new tserver with Master: 700bc936a7384bb89f48a2713b9ad79a (127.0.98.2:43679)
I20260619 08:56:40.889843 8600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53519
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 08:56:40.910254 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41709
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:40.989003 8930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:40.989235 8930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:40.989267 8930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:40.990442 8930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:40.990483 8930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:40.990518 8930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 08:56:40.992065 8930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:40.992340 8930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:40.992547 8930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:40.997344 8938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:40.997349 8936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:40.997581 8930 server_base.cc:1034] running on GCE node
W20260619 08:56:40.997493 8935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:40.997833 8930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:40.998112 8930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:40.999269 8930 hybrid_clock.cc:648] HybridClock initialized: now 1781859400999256 us; error 31 us; skew 500 ppm
Jun 19 08:56:41 dist-test-slave-h4g5 krb5kdc[8555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:41.001760 8930 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 08:56:41.002213 8930 webserver.cc:460] Webserver started at http://127.0.98.3:44395/ using document root <none> and password file <none>
I20260619 08:56:41.002427 8930 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:41.002475 8930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:41.002602 8930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:41.003607 8930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-2/data/instance:
uuid: "5badf2946636412e931df16cab167b54"
format_stamp: "Formatted at 2026-06-19 08:56:41 on dist-test-slave-h4g5"
server_key: "4b5c3d7bd13389728e8596b23574c0a9"
server_key_iv: "d0bdbdc61d47b46761cd7fabbbf16dd4"
server_key_version: "encryptionkey@0"
I20260619 08:56:41.003927 8930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5badf2946636412e931df16cab167b54"
format_stamp: "Formatted at 2026-06-19 08:56:41 on dist-test-slave-h4g5"
server_key: "4b5c3d7bd13389728e8596b23574c0a9"
server_key_iv: "d0bdbdc61d47b46761cd7fabbbf16dd4"
server_key_version: "encryptionkey@0"
I20260619 08:56:41.005265 8930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:56:41.006001 8945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:41.006165 8930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 08:56:41.006246 8930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-2/wal
uuid: "5badf2946636412e931df16cab167b54"
format_stamp: "Formatted at 2026-06-19 08:56:41 on dist-test-slave-h4g5"
server_key: "4b5c3d7bd13389728e8596b23574c0a9"
server_key_iv: "d0bdbdc61d47b46761cd7fabbbf16dd4"
server_key_version: "encryptionkey@0"
I20260619 08:56:41.006315 8930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:41.022588 8930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:41.024825 8930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:41.024961 8930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:41.025187 8930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:41.025482 8930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:41.025521 8930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:41.025550 8930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:41.025574 8930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:41.031040 8930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44927
I20260619 08:56:41.031085 9058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44927 every 8 connection(s)
I20260619 08:56:41.031391 8930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
I20260619 08:56:41.034773 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 8930
I20260619 08:56:41.034878 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-2/wal/instance
I20260619 08:56:41.035058 392 external_mini_cluster.cc:1428] Setting key 61761751fb19a358a4afbc981f5eea83
Jun 19 08:56:41 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:41.040885 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:41.032055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51793 (local address 127.0.98.62:41709)
0619 08:56:41.032153 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:41.032155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:41.032303 (+ 148us) server_negotiation.cc:407] Connection header received
0619 08:56:41.033258 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:41.033259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:41.033271 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:41.033325 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:41.033901 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.034389 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.035788 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.036043 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.036129 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:41.038438 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:41.038440 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:41.038440 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:41.038448 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:41.039756 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:41.040176 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:41.040177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:41.040177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:41.040222 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:41.040398 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:41.040399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:41.040399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:41.040491 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:41.040534 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:41.040763 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 08:56:41.040819 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 08:56:41.041079 9061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:41.032023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.3:51793)
0619 08:56:41.032192 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:41.032226 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:41.033156 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:41.033385 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:41.033388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:41.033643 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:41.033835 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:41.033838 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.034468 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.034469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:41.035715 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:41.035718 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.036152 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.036153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:41.036269 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:41.036805 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:41.036813 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:41.038357 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:41.039832 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:41.039838 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:41.039840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:41.040068 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:41.040261 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:41.040262 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:41.040262 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:41.040357 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:41.040553 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:41.040556 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:41.040668 (+ 112us) client_negotiation.cc:769] Sending connection context
0619 08:56:41.040749 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 08:56:41.040876 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":102,"threads_started":1}
I20260619 08:56:41.041747 9059 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41709
I20260619 08:56:41.041843 9059 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:41.042011 9059 heartbeater.cc:511] Master 127.0.98.62:41709 requested a full tablet report, sending...
I20260619 08:56:41.042356 8600 ts_manager.cc:194] Registered new tserver with Master: 5badf2946636412e931df16cab167b54 (127.0.98.3:44927)
I20260619 08:56:41.042716 8600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51793
I20260619 08:56:41.046685 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 08:56:41.110987 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 8662
I20260619 08:56:41.115361 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 8796
I20260619 08:56:41.119041 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 8930
I20260619 08:56:41.122684 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 8568
I20260619 08:56:41.126560 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41709
--webserver_interface=127.0.98.62
--webserver_port=33629
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41709
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:41.203455 9068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:41.203645 9068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:41.203665 9068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:41.204749 9068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:56:41.204782 9068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:41.204794 9068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:41.204805 9068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:56:41.204818 9068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:56:41.206224 9068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:41.206403 9068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:41.206571 9068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:41.211019 9076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:41.211161 9074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:41.211078 9073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:41.211256 9068 server_base.cc:1034] running on GCE node
I20260619 08:56:41.211478 9068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:41.211747 9068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:41.212883 9068 hybrid_clock.cc:648] HybridClock initialized: now 1781859401212867 us; error 29 us; skew 500 ppm
Jun 19 08:56:41 dist-test-slave-h4g5 krb5kdc[8555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:41.215165 9068 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 08:56:41.215575 9068 webserver.cc:460] Webserver started at http://127.0.98.62:33629/ using document root <none> and password file <none>
I20260619 08:56:41.215761 9068 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:41.215821 9068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:41.217142 9068 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:41.217715 9083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:41.217862 9068 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:41.217940 9068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "e938aee6d2c746f2aa9ee9b88ea2167e"
format_stamp: "Formatted at 2026-06-19 08:56:40 on dist-test-slave-h4g5"
server_key: "00e009499b20c42e1b1c7566d50a8c75"
server_key_iv: "2a3261b3853231755d2a13806a742855"
server_key_version: "encryptionkey@0"
I20260619 08:56:41.218286 9068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:41.235162 9068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:41.236904 9068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:41.237041 9068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:41.240715 9068 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41709
I20260619 08:56:41.240747 9135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41709 every 8 connection(s)
I20260619 08:56:41.241225 9068 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:56:41.242403 9136 sys_catalog.cc:263] Verifying existing consensus state
I20260619 08:56:41.243029 9136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e: Bootstrap starting.
I20260619 08:56:41.250651 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 9068
I20260619 08:56:41.250664 9136 log.cc:824] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:41.250952 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43473
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41709
--webserver_port=33867
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 08:56:41.252676 9136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 08:56:41.252916 9136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e: Bootstrap complete.
I20260619 08:56:41.255894 9136 raft_consensus.cc:348] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e938aee6d2c746f2aa9ee9b88ea2167e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41709 } }
I20260619 08:56:41.256240 9136 raft_consensus.cc:729] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e938aee6d2c746f2aa9ee9b88ea2167e, State: Initialized, Role: FOLLOWER
I20260619 08:56:41.256408 9136 consensus_queue.cc:260] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e938aee6d2c746f2aa9ee9b88ea2167e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41709 } }
I20260619 08:56:41.256480 9136 raft_consensus.cc:388] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:56:41.256527 9136 raft_consensus.cc:482] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:56:41.256577 9136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e [term 1 FOLLOWER]: Advancing to term 2
I20260619 08:56:41.257920 9136 raft_consensus.cc:504] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e938aee6d2c746f2aa9ee9b88ea2167e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41709 } }
I20260619 08:56:41.258033 9136 leader_election.cc:302] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e938aee6d2c746f2aa9ee9b88ea2167e; no voters:
I20260619 08:56:41.258198 9136 leader_election.cc:288] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 08:56:41.258242 9140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e [term 2 FOLLOWER]: Leader election won for term 2
I20260619 08:56:41.258425 9140 raft_consensus.cc:686] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e [term 2 LEADER]: Becoming Leader. State: Replica: e938aee6d2c746f2aa9ee9b88ea2167e, State: Running, Role: LEADER
I20260619 08:56:41.258443 9136 sys_catalog.cc:564] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:56:41.258575 9140 consensus_queue.cc:237] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e938aee6d2c746f2aa9ee9b88ea2167e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41709 } }
I20260619 08:56:41.258867 9141 sys_catalog.cc:455] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e938aee6d2c746f2aa9ee9b88ea2167e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e938aee6d2c746f2aa9ee9b88ea2167e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41709 } } }
I20260619 08:56:41.258936 9141 sys_catalog.cc:458] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:41.259143 9142 sys_catalog.cc:455] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e938aee6d2c746f2aa9ee9b88ea2167e. Latest consensus state: current_term: 2 leader_uuid: "e938aee6d2c746f2aa9ee9b88ea2167e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e938aee6d2c746f2aa9ee9b88ea2167e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41709 } } }
I20260619 08:56:41.259253 9142 sys_catalog.cc:458] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:41.259377 9145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:56:41.259850 9145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:56:41.260008 9145 catalog_manager.cc:1259] Loaded cluster ID: 5d6966e9d98a4fd79417c6004e7d9808
I20260619 08:56:41.260219 9145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:56:41.261029 9145 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:56:41.261183 9145 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e938aee6d2c746f2aa9ee9b88ea2167e: Loaded TSK: 0
I20260619 08:56:41.261377 9145 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 08:56:41.354210 9139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:41.354425 9139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:41.354457 9139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:41.355657 9139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:41.355700 9139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:41.355736 9139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 08:56:41.357254 9139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43473
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33867
--tserver_master_addrs=127.0.98.62:41709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:41.357482 9139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:41.357695 9139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:41.363344 9163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:41.363333 9165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:41.363461 9162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:41.363720 9139 server_base.cc:1034] running on GCE node
I20260619 08:56:41.363919 9139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:41.364197 9139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:41.365345 9139 hybrid_clock.cc:648] HybridClock initialized: now 1781859401365331 us; error 30 us; skew 500 ppm
Jun 19 08:56:41 dist-test-slave-h4g5 krb5kdc[8555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859401, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:41.367880 9139 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 08:56:41.368333 9139 webserver.cc:460] Webserver started at http://127.0.98.1:33867/ using document root <none> and password file <none>
I20260619 08:56:41.368548 9139 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:41.368607 9139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:41.369957 9139 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:41.370532 9172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:41.370709 9139 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:41.370782 9139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-0/wal
uuid: "d3d183f5688940829d692835e9f4b54e"
format_stamp: "Formatted at 2026-06-19 08:56:40 on dist-test-slave-h4g5"
server_key: "2ef1e883ed00538f3047bfed8e537364"
server_key_iv: "d04b4e97a5668f13e4356fe2e8de5400"
server_key_version: "encryptionkey@0"
I20260619 08:56:41.371150 9139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:41.381896 9139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:41.383863 9139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:41.383988 9139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:41.384236 9139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:41.384544 9139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:41.384580 9139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:41.384608 9139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:41.384626 9139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:41.391063 9139 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43473
I20260619 08:56:41.391093 9285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43473 every 8 connection(s)
I20260619 08:56:41.391434 9139 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
I20260619 08:56:41.396454 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 9139
I20260619 08:56:41.396749 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43679
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41709
--webserver_port=46547
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 08:56:41 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:41 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:41.400564 9288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:41.392110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.1:50311)
0619 08:56:41.392290 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:41.392339 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:41.392923 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:41.393524 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:41.393527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:41.394072 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:41.394262 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:41.394267 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.395836 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.395837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:41.396765 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:41.396768 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.397432 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.397433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:41.397553 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:41.398026 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:41.398034 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:41.400371 (+ 2337us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
W20260619 08:56:41.400664 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:41.392212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50311 (local address 127.0.98.62:41709)
0619 08:56:41.393123 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:41.393132 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:41.393168 (+ 36us) server_negotiation.cc:407] Connection header received
0619 08:56:41.393297 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:41.393302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:41.393353 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:41.393476 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:41.394373 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.395750 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.396864 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.397344 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.397444 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:41.400516 (+ 3072us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50311: BlockingRecv error: recv got EOF from 127.0.98.1:50311 (error 108)
Metrics: {"server-negotiator.queue_time_us":861,"thread_start_us":78,"threads_started":1}
W20260619 08:56:41.400830 9286 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41709 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:41709: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 19 08:56:41 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:41 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:41.404942 9288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:41.401041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.1:33631)
0619 08:56:41.401285 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:41.401296 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:41.401342 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:41.401454 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:41.401455 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:41.401576 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:41.401625 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:41.401626 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.402198 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.402199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:41.402698 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:41.402699 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.402989 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.402990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:41.403050 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:41.403419 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:41.403424 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:41.404884 (+ 1460us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226}
W20260619 08:56:41.405020 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:41.401245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33631 (local address 127.0.98.62:41709)
0619 08:56:41.401291 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:41.401293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:41.401328 (+ 35us) server_negotiation.cc:407] Connection header received
0619 08:56:41.401394 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:41.401395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:41.401406 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:41.401446 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:41.401684 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.402136 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.402764 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.402936 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.402987 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:41.404967 (+ 1980us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33631: BlockingRecv error: recv got EOF from 127.0.98.1:33631 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Jun 19 08:56:41 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:41 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:41.409005 9288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:41.405355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.1:58891)
0619 08:56:41.405433 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:41.405442 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:41.405481 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:41.405592 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:41.405593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:41.405711 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:41.405734 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:41.405735 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.406290 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.406291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:41.406769 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:41.406770 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.407064 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.407065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:41.407111 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:41.407456 (+ 345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:41.407460 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:41.408946 (+ 1486us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":65}
W20260619 08:56:41.409075 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:41.405399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58891 (local address 127.0.98.62:41709)
0619 08:56:41.405430 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:41.405432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:41.405469 (+ 37us) server_negotiation.cc:407] Connection header received
0619 08:56:41.405531 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:41.405532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:41.405541 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:41.405585 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:41.405792 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.406229 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.406833 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.407011 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.407061 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:41.409023 (+ 1962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58891: BlockingRecv error: recv got EOF from 127.0.98.1:58891 (error 108)
Metrics: {"server-negotiator.queue_time_us":13}
W20260619 08:56:41.409308 9286 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 08:56:41.479450 9290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:41.479653 9290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:41.479676 9290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:41.480805 9290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:41.480839 9290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:41.480865 9290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 08:56:41.482227 9290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43679
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46547
--tserver_master_addrs=127.0.98.62:41709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:41.482401 9290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:41.482573 9290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:41.487381 9295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:41.487413 9290 server_base.cc:1034] running on GCE node
W20260619 08:56:41.487407 9296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:41.487381 9298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:41.487936 9290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:41.488183 9290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:41.489329 9290 hybrid_clock.cc:648] HybridClock initialized: now 1781859401489318 us; error 31 us; skew 500 ppm
Jun 19 08:56:41 dist-test-slave-h4g5 krb5kdc[8555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859401, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:41.491737 9290 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 08:56:41.492182 9290 webserver.cc:460] Webserver started at http://127.0.98.2:46547/ using document root <none> and password file <none>
I20260619 08:56:41.492374 9290 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:41.492429 9290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:41.493681 9290 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260619 08:56:41.494166 9305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:41.494292 9290 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:41.494367 9290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-1/wal
uuid: "700bc936a7384bb89f48a2713b9ad79a"
format_stamp: "Formatted at 2026-06-19 08:56:40 on dist-test-slave-h4g5"
server_key: "0038410e28547783394bfa0eba7d46c4"
server_key_iv: "1a370c23e888f271a260ab40bc556b6f"
server_key_version: "encryptionkey@0"
I20260619 08:56:41.494683 9290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:41.507269 9290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:41.509343 9290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:41.509472 9290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:41.509682 9290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:41.509955 9290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:41.509987 9290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:41.510015 9290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:41.510038 9290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:41.515606 9290 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43679
I20260619 08:56:41.515671 9418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43679 every 8 connection(s)
I20260619 08:56:41.515973 9290 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 08:56:41 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:41 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:41.522352 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 9290
I20260619 08:56:41.522588 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44927
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41709
--webserver_port=44395
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:41.522668 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:41.516680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33411 (local address 127.0.98.62:41709)
0619 08:56:41.516740 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:41.516742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:41.516929 (+ 187us) server_negotiation.cc:407] Connection header received
0619 08:56:41.517374 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:41.517375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:41.517387 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:41.517434 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:41.518282 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.518760 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.519703 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.519872 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.519917 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:41.522614 (+ 2697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33411: BlockingRecv error: recv got EOF from 127.0.98.2:33411 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 08:56:41.522737 9421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:41.516669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.2:33411)
0619 08:56:41.516816 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:41.516851 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:41.517301 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:41.517488 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:41.517491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:41.518018 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:41.518207 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:41.518212 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.518836 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.518837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:41.519624 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:41.519626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.519928 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.519928 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:41.520019 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:41.520474 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:41.520482 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:41.522533 (+ 2051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
W20260619 08:56:41.522921 9419 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41709 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:41709: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 19 08:56:41 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:41 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:41.527410 9421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:41.523106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.2:57017)
0619 08:56:41.523199 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:41.523209 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:41.523259 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:41.523365 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:41.523365 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:41.523490 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:41.523538 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:41.523539 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.524373 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.524373 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:41.524862 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:41.524863 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.525174 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.525174 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:41.525234 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:41.525597 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:41.525600 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:41.527346 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":77}
W20260619 08:56:41.527477 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:41.523153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57017 (local address 127.0.98.62:41709)
0619 08:56:41.523195 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:41.523196 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:41.523245 (+ 49us) server_negotiation.cc:407] Connection header received
0619 08:56:41.523313 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:41.523314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:41.523321 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:41.523360 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:41.523597 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.524303 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.524940 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.525117 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.525178 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:41.527426 (+ 2248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57017: BlockingRecv error: recv got EOF from 127.0.98.2:57017 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Jun 19 08:56:41 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:41 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:41.531519 9421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:41.527727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.2:54509)
0619 08:56:41.527758 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:41.527767 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:41.527801 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:41.527931 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:41.527932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:41.528044 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:41.528069 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:41.528070 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.528641 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.528642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:41.529114 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:41.529115 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.529407 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.529408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:41.529454 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:41.529841 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:41.529845 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:41.531457 (+ 1612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":16}
W20260619 08:56:41.531638 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:41.527795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54509 (local address 127.0.98.62:41709)
0619 08:56:41.527848 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:41.527850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:41.527857 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:56:41.527874 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:41.527875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:41.527882 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:41.527922 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:41.528134 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.528575 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.529184 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.529355 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.529405 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:41.531537 (+ 2132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54509: BlockingRecv error: recv got EOF from 127.0.98.2:54509 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260619 08:56:41.531635 9419 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 08:56:41.602607 9422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:41.602805 9422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:41.602826 9422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:41.603974 9422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:41.604014 9422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:41.604043 9422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 08:56:41.605454 9422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44927
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44395
--tserver_master_addrs=127.0.98.62:41709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:41.605669 9422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:41.605870 9422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:41.610509 9428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:41.610527 9427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:41.610562 9430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:41.610718 9422 server_base.cc:1034] running on GCE node
I20260619 08:56:41.610922 9422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:41.611155 9422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:41.612294 9422 hybrid_clock.cc:648] HybridClock initialized: now 1781859401612274 us; error 35 us; skew 500 ppm
Jun 19 08:56:41 dist-test-slave-h4g5 krb5kdc[8555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859401, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:41.614609 9422 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 08:56:41.615026 9422 webserver.cc:460] Webserver started at http://127.0.98.3:44395/ using document root <none> and password file <none>
I20260619 08:56:41.615203 9422 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:41.615253 9422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:41.616617 9422 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:41.617228 9437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:41.617386 9422 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:41.617465 9422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-2/wal
uuid: "5badf2946636412e931df16cab167b54"
format_stamp: "Formatted at 2026-06-19 08:56:41 on dist-test-slave-h4g5"
server_key: "4b5c3d7bd13389728e8596b23574c0a9"
server_key_iv: "d0bdbdc61d47b46761cd7fabbbf16dd4"
server_key_version: "encryptionkey@0"
I20260619 08:56:41.617810 9422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:41.632925 9422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:41.634629 9422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:41.634743 9422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:41.634923 9422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:41.635195 9422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:41.635227 9422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:41.635250 9422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:41.635273 9422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:41.640749 9422 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44927
I20260619 08:56:41.640821 9550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44927 every 8 connection(s)
I20260619 08:56:41.641114 9422 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 08:56:41 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:41 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:41.647534 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 9422
W20260619 08:56:41.647739 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:41.641764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40239 (local address 127.0.98.62:41709)
0619 08:56:41.641825 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:41.641828 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:41.642061 (+ 233us) server_negotiation.cc:407] Connection header received
0619 08:56:41.642486 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:41.642487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:41.642498 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:41.642570 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:41.643456 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.643945 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.644932 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.645110 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.645160 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:41.647682 (+ 2522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40239: BlockingRecv error: recv got EOF from 127.0.98.3:40239 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 08:56:41.647742 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:41.641753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.3:40239)
0619 08:56:41.641903 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:41.641945 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:41.642410 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:41.642613 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:41.642617 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:41.643156 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:41.643356 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:41.643360 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.644018 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.644019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:41.644866 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:41.644868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.645166 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.645166 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:41.645278 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:41.645693 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:41.645700 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:41.647563 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
W20260619 08:56:41.648154 9551 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41709 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:41709: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 19 08:56:41 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:41 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:41.652486 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:41.648357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.3:43289)
0619 08:56:41.648497 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:41.648506 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:41.648547 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:41.648676 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:41.648677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:41.648803 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:41.648844 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:41.648845 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.649442 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.649443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:41.649935 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:41.649936 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.650242 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.650243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:41.650302 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:41.650673 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:41.650678 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:41.652421 (+ 1743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125}
W20260619 08:56:41.652588 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:41.648399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43289 (local address 127.0.98.62:41709)
0619 08:56:41.648495 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:41.648497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:41.648534 (+ 37us) server_negotiation.cc:407] Connection header received
0619 08:56:41.648594 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:41.648594 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:41.648604 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:41.648669 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:41.648907 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.649380 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.650000 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.650188 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.650240 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:41.652529 (+ 2289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43289: BlockingRecv error: recv got EOF from 127.0.98.3:43289 (error 108)
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":65,"threads_started":1}
I20260619 08:56:41.654291 9289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:41.648019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43266 (local address 127.0.98.62:41709)
0619 08:56:41.648063 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:41.648065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:41.648264 (+ 199us) server_negotiation.cc:407] Connection header received
0619 08:56:41.648337 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:41.648338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:41.648348 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:41.648391 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:41.648972 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.649417 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.650140 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.650316 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.650366 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:41.651451 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:41.651457 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:41.651457 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:41.651469 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:41.653107 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:41.653416 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:41.653419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:41.653420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:41.653506 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:41.653670 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:41.653671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:41.653671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:41.653886 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:41.654014 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:41.654116 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 08:56:41.654217 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Jun 19 08:56:41 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:41 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:41.656625 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:41.652699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.3:32941)
0619 08:56:41.652831 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:41.652841 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:41.652885 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:41.653005 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:41.653006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:41.653134 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:41.653179 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:41.653180 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.653754 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.653755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:41.654297 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:41.654299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.654622 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.654623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:41.654684 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:41.655023 (+ 339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:41.655027 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:41.656570 (+ 1543us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
W20260619 08:56:41.656687 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:41.652882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32941 (local address 127.0.98.62:41709)
0619 08:56:41.652920 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:41.652922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:41.652928 (+ 6us) server_negotiation.cc:407] Connection header received
0619 08:56:41.652944 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:41.652945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:41.652954 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:41.652993 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:41.653251 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.653693 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.654370 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:41.654540 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:41.654584 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:41.656640 (+ 2056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:32941: BlockingRecv error: recv got EOF from 127.0.98.3:32941 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 08:56:41.656754 9551 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 08:56:42 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:42 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:42.414089 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:42.409633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.1:35717)
0619 08:56:42.409767 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:42.409778 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:42.409826 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:42.409941 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:42.409942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:42.410085 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:42.410132 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:42.410134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:42.410751 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:42.410752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:42.411299 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:42.411302 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:42.411599 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:42.411600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:42.411636 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:42.412173 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:42.412178 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:42.414015 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
W20260619 08:56:42.414188 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:42.409703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35717 (local address 127.0.98.62:41709)
0619 08:56:42.409809 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:42.409811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:42.409819 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:56:42.409867 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:42.409868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:42.409877 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:42.409930 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:42.410180 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:42.410666 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:42.411366 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:42.411547 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:42.411593 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:42.414119 (+ 2526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35717: BlockingRecv error: recv got EOF from 127.0.98.1:35717 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":48,"threads_started":1}
Jun 19 08:56:42 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:42 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:42.536331 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:42.532041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.2:41489)
0619 08:56:42.532201 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:42.532211 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:42.532279 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:42.532439 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:42.532440 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:42.532591 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:42.532640 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:42.532641 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:42.533261 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:42.533261 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:42.533795 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:42.533796 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:42.534101 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:42.534101 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:42.534135 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:42.534541 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:42.534545 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:42.536250 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":95,"threads_started":1}
W20260619 08:56:42.536389 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:42.532128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41489 (local address 127.0.98.62:41709)
0619 08:56:42.532186 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:42.532188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:42.532272 (+ 84us) server_negotiation.cc:407] Connection header received
0619 08:56:42.532337 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:42.532338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:42.532348 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:42.532395 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:42.532720 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:42.533196 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:42.533863 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:42.534047 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:42.534099 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:42.536340 (+ 2241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41489: BlockingRecv error: recv got EOF from 127.0.98.2:41489 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Jun 19 08:56:42 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:42 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:42.661785 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:42.657255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.3:49633)
0619 08:56:42.657496 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:42.657518 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:42.657596 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:42.657720 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:42.657721 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:42.657926 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:42.658000 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:42.658003 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:42.658697 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:42.658698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:42.659256 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:42.659259 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:42.659608 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:42.659608 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:42.659674 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:42.660136 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:42.660140 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:42.661727 (+ 1587us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":91,"threads_started":1}
W20260619 08:56:42.661875 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:42.657376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49633 (local address 127.0.98.62:41709)
0619 08:56:42.657454 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:42.657456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:42.657588 (+ 132us) server_negotiation.cc:407] Connection header received
0619 08:56:42.657644 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:42.657645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:42.657657 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:42.657704 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:42.658064 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:42.658632 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:42.659342 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:42.659539 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:42.659601 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:42.661810 (+ 2209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49633: BlockingRecv error: recv got EOF from 127.0.98.3:49633 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 19 08:56:43 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:43 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:43.419098 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:43.414598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.1:38995)
0619 08:56:43.414752 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:43.414764 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:43.414827 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:43.414976 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:43.414977 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:43.415169 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:43.415230 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:43.415232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:43.415937 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:43.415938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:43.416553 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:43.416555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:43.416926 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:43.416927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:43.416970 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:43.417409 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:43.417414 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:43.419029 (+ 1615us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":58,"threads_started":1}
W20260619 08:56:43.419217 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:43.414655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38995 (local address 127.0.98.62:41709)
0619 08:56:43.414764 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:43.414766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:43.414822 (+ 56us) server_negotiation.cc:407] Connection header received
0619 08:56:43.414874 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:43.414875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:43.414885 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:43.414964 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:43.415331 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:43.415853 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:43.416641 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:43.416858 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:43.416920 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:43.419147 (+ 2227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38995: BlockingRecv error: recv got EOF from 127.0.98.1:38995 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":59,"threads_started":1}
Jun 19 08:56:43 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:43 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:43.541330 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:43.536868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.2:37767)
0619 08:56:43.537031 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:43.537041 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:43.537103 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:43.537253 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:43.537253 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:43.537424 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:43.537477 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:43.537478 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:43.538114 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:43.538116 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:43.538680 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:43.538682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:43.539074 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:43.539075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:43.539125 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:43.539566 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:43.539570 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:43.541260 (+ 1690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":81,"threads_started":1}
W20260619 08:56:43.541450 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:43.536979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37767 (local address 127.0.98.62:41709)
0619 08:56:43.537036 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:43.537037 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:43.537097 (+ 60us) server_negotiation.cc:407] Connection header received
0619 08:56:43.537150 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:43.537151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:43.537176 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:43.537229 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:43.537533 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:43.538026 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:43.538756 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:43.538952 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:43.539004 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:43.541383 (+ 2379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37767: BlockingRecv error: recv got EOF from 127.0.98.2:37767 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 19 08:56:43 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:43 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:43.666540 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:43.662289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.3:36335)
0619 08:56:43.662420 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:43.662432 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:43.662515 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:43.662650 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:43.662655 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:43.662856 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:43.662917 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:43.662920 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:43.663548 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:43.663549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:43.664074 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:43.664076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:43.664406 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:43.664407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:43.664470 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:43.664929 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:43.664933 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:43.666461 (+ 1528us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":58,"threads_started":1}
W20260619 08:56:43.666604 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:43.662442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36335 (local address 127.0.98.62:41709)
0619 08:56:43.662487 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:43.662489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:43.662497 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:56:43.662570 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:43.662571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:43.662585 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:43.662634 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:43.662987 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:43.663480 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:43.664163 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:43.664349 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:43.664402 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:43.666551 (+ 2149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36335: BlockingRecv error: recv got EOF from 127.0.98.3:36335 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Jun 19 08:56:44 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:44 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:44.427595 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:44.419532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.1:34279)
0619 08:56:44.419654 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:44.419664 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:44.419711 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:44.419852 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:44.419853 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:44.420003 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:44.420052 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:44.420053 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:44.420795 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:44.420796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:44.421337 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:44.421339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:44.421728 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:44.421729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:44.421776 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:44.425820 (+ 4044us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:44.425825 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:44.427530 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":58,"threads_started":1}
W20260619 08:56:44.427686 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:44.419599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34279 (local address 127.0.98.62:41709)
0619 08:56:44.419706 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:44.419708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:44.419717 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:56:44.419763 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:44.419764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:44.419776 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:44.419831 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:44.420131 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:44.420725 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:44.421437 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:44.421660 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:44.421709 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:44.427621 (+ 5912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34279: BlockingRecv error: recv got EOF from 127.0.98.1:34279 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":57,"threads_started":1}
Jun 19 08:56:44 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:44 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:44.546203 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:44.541802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.2:37685)
0619 08:56:44.541920 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:44.541934 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:44.541987 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:44.542109 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:44.542110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:44.542261 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:44.542307 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:44.542309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:44.542987 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:44.542987 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:44.543527 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:44.543529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:44.543943 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:44.543943 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:44.543987 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:44.544411 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:44.544415 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:44.546144 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":57,"threads_started":1}
W20260619 08:56:44.546332 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:44.541872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37685 (local address 127.0.98.62:41709)
0619 08:56:44.541956 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:44.541958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:44.541971 (+ 13us) server_negotiation.cc:407] Connection header received
0619 08:56:44.542035 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:44.542036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:44.542048 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:44.542096 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:44.542397 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:44.542919 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:44.543633 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:44.543871 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:44.543940 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:44.546265 (+ 2325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37685: BlockingRecv error: recv got EOF from 127.0.98.2:37685 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 08:56:44 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:44 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:44.671239 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:44.666978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.3:55451)
0619 08:56:44.667126 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:44.667137 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:44.667210 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:44.667338 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:44.667340 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:44.667505 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:44.667550 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:44.667553 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:44.668219 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:44.668220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:44.668759 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:44.668762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:44.669084 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:44.669085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:44.669138 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:44.669585 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:44.669589 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:44.671163 (+ 1574us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":80,"threads_started":1}
W20260619 08:56:44.671324 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:44.667064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55451 (local address 127.0.98.62:41709)
0619 08:56:44.667131 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:44.667133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:44.667201 (+ 68us) server_negotiation.cc:407] Connection header received
0619 08:56:44.667264 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:44.667265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:44.667279 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:44.667327 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:44.667627 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:44.668156 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:44.668839 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:44.669026 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:44.669081 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:44.671270 (+ 2189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55451: BlockingRecv error: recv got EOF from 127.0.98.3:55451 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 19 08:56:45 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:45 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:45.432822 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:45.428102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.1:51671)
0619 08:56:45.428232 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:45.428245 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:45.428301 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:45.428424 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:45.428425 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:45.428570 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:45.428617 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:45.428619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:45.429276 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:45.429278 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:45.430101 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:45.430104 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:45.430469 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:45.430470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:45.430551 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:45.430992 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:45.430997 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:45.432745 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":58,"threads_started":1}
W20260619 08:56:45.432902 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:45.428152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51671 (local address 127.0.98.62:41709)
0619 08:56:45.428283 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:45.428285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:45.428296 (+ 11us) server_negotiation.cc:407] Connection header received
0619 08:56:45.428347 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:45.428347 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:45.428357 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:45.428414 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:45.428683 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:45.429183 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:45.430195 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:45.430394 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:45.430455 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:45.432841 (+ 2386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51671: BlockingRecv error: recv got EOF from 127.0.98.1:51671 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
Jun 19 08:56:45 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:45 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:45.551328 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:45.546684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.2:50017)
0619 08:56:45.546823 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:45.546833 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:45.546894 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:45.547067 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:45.547069 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:45.547230 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:45.547278 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:45.547280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:45.547911 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:45.547912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:45.548487 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:45.548489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:45.548901 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:45.548902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:45.548959 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:45.549486 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:45.549491 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:45.551267 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
W20260619 08:56:45.551462 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:45.546779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50017 (local address 127.0.98.62:41709)
0619 08:56:45.546836 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:45.546838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:45.546890 (+ 52us) server_negotiation.cc:407] Connection header received
0619 08:56:45.546957 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:45.546958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:45.546968 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:45.547010 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:45.547348 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:45.547839 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:45.548581 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:45.548791 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:45.548858 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:45.551396 (+ 2538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50017: BlockingRecv error: recv got EOF from 127.0.98.2:50017 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 19 08:56:45 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:45 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:45.676360 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:45.671862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.3:40535)
0619 08:56:45.672024 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:45.672035 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:45.672137 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:45.672323 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:45.672324 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:45.672487 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:45.672571 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:45.672573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:45.673259 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:45.673259 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:45.673864 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:45.673866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:45.674186 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:45.674187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:45.674223 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:45.674638 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:45.674643 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:45.676293 (+ 1650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":61,"threads_started":1}
W20260619 08:56:45.676445 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:45.671979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40535 (local address 127.0.98.62:41709)
0619 08:56:45.672037 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:45.672038 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:45.672110 (+ 72us) server_negotiation.cc:407] Connection header received
0619 08:56:45.672183 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:45.672184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:45.672195 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:45.672254 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:45.672693 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:45.673189 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:45.673933 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:45.674130 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:45.674185 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:45.676385 (+ 2200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40535: BlockingRecv error: recv got EOF from 127.0.98.3:40535 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 19 08:56:46 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:46 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:46.437765 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:46.433301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.1:41445)
0619 08:56:46.433457 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:46.433467 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:46.433525 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:46.433640 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:46.433641 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:46.433788 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:46.433832 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:46.433834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:46.434498 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:46.434499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:46.435055 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:46.435057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:46.435406 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:46.435407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:46.435458 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:46.435961 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:46.435966 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:46.437705 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":88,"threads_started":1}
W20260619 08:56:46.437878 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:46.433375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41445 (local address 127.0.98.62:41709)
0619 08:56:46.433483 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:46.433485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:46.433512 (+ 27us) server_negotiation.cc:407] Connection header received
0619 08:56:46.433576 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:46.433576 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:46.433586 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:46.433632 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:46.433925 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:46.434424 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:46.435130 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:46.435324 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:46.435375 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:46.437806 (+ 2431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41445: BlockingRecv error: recv got EOF from 127.0.98.1:41445 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":55,"threads_started":1}
Jun 19 08:56:46 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:46 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:46.556524 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:46.551878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.2:51091)
0619 08:56:46.552019 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:46.552032 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:46.552126 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:46.552249 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:46.552249 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:46.552404 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:46.552456 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:46.552458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:46.553181 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:46.553182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:46.553717 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:46.553718 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:46.554075 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:46.554076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:46.554141 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:46.554591 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:46.554597 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:46.556452 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
W20260619 08:56:46.556643 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:46.552046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51091 (local address 127.0.98.62:41709)
0619 08:56:46.552130 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:46.552132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:46.552141 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:56:46.552172 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:46.552173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:46.552183 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:46.552231 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:46.552539 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:46.553080 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:46.553786 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:46.553986 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:46.554043 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:46.556578 (+ 2535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51091: BlockingRecv error: recv got EOF from 127.0.98.2:51091 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 08:56:46 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:46 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:46.681429 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:46.676852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.3:51909)
0619 08:56:46.676984 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:46.676995 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:46.677054 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:46.677169 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:46.677170 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:46.677342 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:46.677389 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:46.677391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:46.678229 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:46.678231 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:46.678837 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:46.678839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:46.679182 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:46.679183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:46.679218 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:46.679630 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:46.679635 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:46.681359 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
W20260619 08:56:46.681512 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:46.676967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51909 (local address 127.0.98.62:41709)
0619 08:56:46.677034 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:46.677036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:46.677045 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:56:46.677106 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:46.677107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:46.677118 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:46.677163 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:46.677482 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:46.678164 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:46.678928 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:46.679123 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:46.679179 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:46.681448 (+ 2269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51909: BlockingRecv error: recv got EOF from 127.0.98.3:51909 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 19 08:56:47 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:47 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:47.442999 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:47.438253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.1:40417)
0619 08:56:47.438370 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:47.438381 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:47.438437 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:47.438593 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:47.438595 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:47.438784 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:47.438836 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:47.438838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:47.439518 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:47.439519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:47.440166 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:47.440169 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:47.440552 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:47.440553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:47.440611 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:47.441098 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:47.441103 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:47.442916 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":67,"threads_started":1}
W20260619 08:56:47.443099 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:47.438313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40417 (local address 127.0.98.62:41709)
0619 08:56:47.438433 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:47.438435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:47.438445 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:56:47.438493 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:47.438494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:47.438508 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:47.438580 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:47.438916 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:47.439432 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:47.440266 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:47.440473 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:47.440537 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:47.443032 (+ 2495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40417: BlockingRecv error: recv got EOF from 127.0.98.1:40417 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
Jun 19 08:56:47 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:47 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:47.561947 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:47.557152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.2:58383)
0619 08:56:47.557300 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:47.557310 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:47.557365 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:47.557504 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:47.557504 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:47.557666 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:47.557718 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:47.557720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:47.558519 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:47.558520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:47.559088 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:47.559091 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:47.559564 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:47.559565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:47.559622 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:47.560111 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:47.560117 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:47.561858 (+ 1741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
W20260619 08:56:47.562054 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:47.557259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58383 (local address 127.0.98.62:41709)
0619 08:56:47.557334 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:47.557336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:47.557351 (+ 15us) server_negotiation.cc:407] Connection header received
0619 08:56:47.557420 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:47.557421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:47.557432 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:47.557491 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:47.557783 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:47.558384 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:47.559227 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:47.559490 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:47.559550 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:47.561987 (+ 2437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58383: BlockingRecv error: recv got EOF from 127.0.98.2:58383 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 08:56:47 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:47 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:47.686232 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:47.681903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.3:50487)
0619 08:56:47.682050 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:47.682063 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:47.682140 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:47.682276 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:47.682277 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:47.682459 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:47.682506 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:47.682508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:47.683171 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:47.683173 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:47.683698 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:47.683700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:47.684029 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:47.684030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:47.684111 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:47.684528 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:47.684532 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:47.686156 (+ 1624us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
W20260619 08:56:47.686305 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:47.682005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50487 (local address 127.0.98.62:41709)
0619 08:56:47.682087 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:47.682089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:47.682119 (+ 30us) server_negotiation.cc:407] Connection header received
0619 08:56:47.682200 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:47.682201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:47.682215 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:47.682261 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:47.682586 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:47.683092 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:47.683775 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:47.683969 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:47.684023 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:47.686252 (+ 2229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50487: BlockingRecv error: recv got EOF from 127.0.98.3:50487 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 08:56:48 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:48 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:48.447680 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:48.443482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.1:59769)
0619 08:56:48.443626 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:48.443637 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:48.443691 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:48.443811 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:48.443812 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:48.443969 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:48.444014 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:48.444016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:48.444673 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:48.444673 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:48.445206 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:48.445208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:48.445568 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:48.445568 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:48.445610 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:48.446023 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:48.446027 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:48.447622 (+ 1595us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":76,"threads_started":1}
W20260619 08:56:48.447789 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:48.443586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59769 (local address 127.0.98.62:41709)
0619 08:56:48.443692 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:48.443694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:48.443719 (+ 25us) server_negotiation.cc:407] Connection header received
0619 08:56:48.443738 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:48.443739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:48.443748 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:48.443793 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:48.444070 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:48.444583 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:48.445303 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:48.445506 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:48.445561 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:48.447714 (+ 2153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59769: BlockingRecv error: recv got EOF from 127.0.98.1:59769 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":50,"threads_started":1}
Jun 19 08:56:48 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:48 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:48.566946 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:48.562432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.2:42657)
0619 08:56:48.562566 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:48.562578 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:48.562647 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:48.562768 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:48.562769 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:48.562936 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:48.562986 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:48.562989 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:48.563641 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:48.563642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:48.564215 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:48.564217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:48.564536 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:48.564537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:48.564587 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:48.565061 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:48.565066 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:48.566854 (+ 1788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
W20260619 08:56:48.567037 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:48.562507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42657 (local address 127.0.98.62:41709)
0619 08:56:48.562578 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:48.562580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:48.562627 (+ 47us) server_negotiation.cc:407] Connection header received
0619 08:56:48.562706 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:48.562706 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:48.562717 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:48.562765 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:48.563062 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:48.563569 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:48.564289 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:48.564481 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:48.564532 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:48.566950 (+ 2418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42657: BlockingRecv error: recv got EOF from 127.0.98.2:42657 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 08:56:48 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:48 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:48.691126 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:48.686732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.3:35777)
0619 08:56:48.686865 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:48.686875 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:48.686933 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:48.687057 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:48.687058 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:48.687205 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:48.687252 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:48.687254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:48.687851 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:48.687852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:48.688463 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:48.688466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:48.688779 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:48.688780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:48.688819 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:48.689314 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:48.689319 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:48.691067 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":69,"threads_started":1}
W20260619 08:56:48.691217 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:48.686803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35777 (local address 127.0.98.62:41709)
0619 08:56:48.686920 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:48.686922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:48.686932 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:56:48.686993 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:48.686994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:48.687005 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:48.687045 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:48.687306 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:48.687793 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:48.688537 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:48.688724 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:48.688778 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:48.691154 (+ 2376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35777: BlockingRecv error: recv got EOF from 127.0.98.3:35777 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 08:56:49 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:49 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:49.457849 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:49.448140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.1:60097)
0619 08:56:49.448257 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:49.448269 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:49.448317 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:49.448542 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:49.448543 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:49.448717 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:49.448766 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:49.448768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:49.449711 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:49.449711 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:49.450560 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:49.450563 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:49.451040 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:49.451040 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:49.451093 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:49.455791 (+ 4698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:49.455796 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:49.457778 (+ 1982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":62,"threads_started":1}
W20260619 08:56:49.457978 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:49.448257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60097 (local address 127.0.98.62:41709)
0619 08:56:49.448392 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:49.448394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:49.448404 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:56:49.448446 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:49.448447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:49.448460 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:49.448538 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:49.448840 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:49.449629 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:49.450650 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:49.450973 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:49.451042 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:49.457885 (+ 6843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60097: BlockingRecv error: recv got EOF from 127.0.98.1:60097 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
Jun 19 08:56:49 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:49 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:49.572175 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:49.567465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.2:46243)
0619 08:56:49.567605 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:49.567617 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:49.567667 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:49.567803 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:49.567804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:49.567954 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:49.568001 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:49.568002 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:49.568774 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:49.568775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:49.569302 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:49.569303 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:49.569649 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:49.569650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:49.569685 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:49.570158 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:49.570164 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:49.572058 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":60,"threads_started":1}
W20260619 08:56:49.572274 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:49.567612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46243 (local address 127.0.98.62:41709)
0619 08:56:49.567670 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:49.567671 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:49.567683 (+ 12us) server_negotiation.cc:407] Connection header received
0619 08:56:49.567733 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:49.567734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:49.567744 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:49.567792 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:49.568057 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:49.568705 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:49.569389 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:49.569589 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:49.569647 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:49.572215 (+ 2568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46243: BlockingRecv error: recv got EOF from 127.0.98.2:46243 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 19 08:56:49 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:49 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:49.695879 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:49.691643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.3:42959)
0619 08:56:49.691804 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:49.691814 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:49.691871 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:49.691980 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:49.691980 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:49.692210 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:49.692257 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:49.692259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:49.692975 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:49.692976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:49.693526 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:49.693528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:49.693846 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:49.693847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:49.693895 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:49.694338 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:49.694343 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:49.695821 (+ 1478us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":76,"threads_started":1}
W20260619 08:56:49.695984 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:49.691687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42959 (local address 127.0.98.62:41709)
0619 08:56:49.691729 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:49.691731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:49.691861 (+ 130us) server_negotiation.cc:407] Connection header received
0619 08:56:49.691917 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:49.691918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:49.691930 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:49.691974 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:49.692357 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:49.692904 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:49.693589 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:49.693768 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:49.693815 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:49.695916 (+ 2101us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42959: BlockingRecv error: recv got EOF from 127.0.98.3:42959 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Jun 19 08:56:50 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:50 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:50.462993 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:50.458333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.1:34675)
0619 08:56:50.458452 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:50.458462 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:50.458515 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:50.458628 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:50.458629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:50.458829 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:50.458878 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:50.458880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:50.459710 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:50.459711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:50.460367 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:50.460370 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:50.460728 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:50.460728 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:50.460765 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:50.461232 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:50.461237 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:50.462934 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":66,"threads_started":1}
W20260619 08:56:50.463091 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:50.458414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34675 (local address 127.0.98.62:41709)
0619 08:56:50.458517 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:50.458518 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:50.458547 (+ 29us) server_negotiation.cc:407] Connection header received
0619 08:56:50.458565 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:50.458566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:50.458574 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:50.458616 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:50.458991 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:50.459607 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:50.460457 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:50.460662 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:50.460720 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:50.463025 (+ 2305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34675: BlockingRecv error: recv got EOF from 127.0.98.1:34675 (error 108)
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":46,"threads_started":1}
Jun 19 08:56:50 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:50 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:50.577495 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:50.572777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.2:49825)
0619 08:56:50.572899 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:50.572912 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:50.572981 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:50.573176 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:50.573177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:50.573342 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:50.573397 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:50.573399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:50.574068 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:50.574068 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:50.574740 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:50.574743 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:50.575089 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:50.575090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:50.575125 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:50.575604 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:50.575608 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:50.577416 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":56,"threads_started":1}
W20260619 08:56:50.577564 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:50.572886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49825 (local address 127.0.98.62:41709)
0619 08:56:50.572982 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:50.572984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:50.573001 (+ 17us) server_negotiation.cc:407] Connection header received
0619 08:56:50.573025 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:50.573059 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:50.573071 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:50.573128 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:50.573489 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:50.574002 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:50.574833 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:50.575026 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:50.575095 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:50.577502 (+ 2407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49825: BlockingRecv error: recv got EOF from 127.0.98.2:49825 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 08:56:50 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:50 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:50.700883 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:50.696407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.3:52771)
0619 08:56:50.696534 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:50.696546 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:50.696665 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:50.696822 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:50.696824 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:50.697035 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:50.697092 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:50.697094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:50.697717 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:50.697718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:50.698331 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:50.698333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:50.698647 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:50.698648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:50.698685 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:50.699133 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:50.699139 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:50.700825 (+ 1686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
W20260619 08:56:50.700970 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:50.696525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52771 (local address 127.0.98.62:41709)
0619 08:56:50.696600 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:50.696603 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:50.696646 (+ 43us) server_negotiation.cc:407] Connection header received
0619 08:56:50.696728 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:50.696729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:50.696741 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:50.696806 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:50.697162 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:50.697651 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:50.698403 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:50.698600 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:50.698646 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:50.700910 (+ 2264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52771: BlockingRecv error: recv got EOF from 127.0.98.3:52771 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 08:56:51 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:51 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:51.467911 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:51.463496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.1:38793)
0619 08:56:51.463648 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:51.463659 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:51.463724 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:51.463853 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:51.463854 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:51.464008 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:51.464055 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:51.464057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:51.464729 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:51.464729 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:51.465254 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:51.465255 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:51.465560 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:51.465561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:51.465597 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:51.466010 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:51.466014 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:51.467822 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":88,"threads_started":1}
W20260619 08:56:51.467963 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:51.463569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38793 (local address 127.0.98.62:41709)
0619 08:56:51.463670 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:51.463672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:51.463722 (+ 50us) server_negotiation.cc:407] Connection header received
0619 08:56:51.463778 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:51.463778 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:51.463788 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:51.463835 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:51.464143 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:51.464668 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:51.465310 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:51.465511 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:51.465555 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:51.467903 (+ 2348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38793: BlockingRecv error: recv got EOF from 127.0.98.1:38793 (error 108)
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":51,"threads_started":1}
Jun 19 08:56:51 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:51 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:51.582149 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:51.577927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.2:49863)
0619 08:56:51.578068 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:51.578079 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:51.578138 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:51.578262 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:51.578263 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:51.578431 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:51.578479 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:51.578480 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:51.579089 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:51.579089 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:51.579625 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:51.579626 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:51.579961 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:51.579962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:51.580020 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:51.580503 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:51.580508 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:51.582089 (+ 1581us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":81,"threads_started":1}
W20260619 08:56:51.582235 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:51.577982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49863 (local address 127.0.98.62:41709)
0619 08:56:51.578057 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:51.578058 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:51.578131 (+ 73us) server_negotiation.cc:407] Connection header received
0619 08:56:51.578207 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:51.578207 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:51.578217 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:51.578256 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:51.578528 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:51.579029 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:51.579680 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:51.579852 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:51.579908 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:51.582177 (+ 2269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49863: BlockingRecv error: recv got EOF from 127.0.98.2:49863 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 08:56:51 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:51 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:51.705603 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:51.701389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.3:52181)
0619 08:56:51.701560 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:51.701573 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:51.701642 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:51.701762 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:51.701763 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:51.701918 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:51.701967 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:51.701969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:51.702565 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:51.702566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:51.703082 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:51.703083 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:51.703350 (+ 267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:51.703351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:51.703402 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:51.703814 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:51.703818 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:51.705528 (+ 1710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":85,"threads_started":1}
W20260619 08:56:51.705691 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:51.701530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52181 (local address 127.0.98.62:41709)
0619 08:56:51.701595 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:51.701597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:51.701625 (+ 28us) server_negotiation.cc:407] Connection header received
0619 08:56:51.701690 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:51.701691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:51.701702 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:51.701751 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:51.702016 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:51.702512 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:51.703138 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:51.703306 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:51.703347 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:51.705628 (+ 2281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52181: BlockingRecv error: recv got EOF from 127.0.98.3:52181 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 19 08:56:52 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:52 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:52.472857 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:52.468416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.1:45883)
0619 08:56:52.468556 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:52.468568 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:52.468637 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:52.468758 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:52.468759 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:52.468916 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:52.468963 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:52.468965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:52.469581 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:52.469582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:52.470100 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:52.470101 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:52.470435 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:52.470436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:52.470500 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:52.470959 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:52.470964 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:52.472799 (+ 1835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
W20260619 08:56:52.472922 9598 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:52.468494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45883 (local address 127.0.98.62:41709)
0619 08:56:52.468647 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:52.468649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:52.468657 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:56:52.468680 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:52.468681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:52.468691 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:52.468747 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:52.469017 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:52.469520 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:52.470162 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:52.470351 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:52.470412 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:52.472866 (+ 2454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45883: BlockingRecv error: recv got EOF from 127.0.98.1:45883 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":77,"threads_started":1}
Jun 19 08:56:52 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:52 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:52.587184 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:52.582651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.2:55593)
0619 08:56:52.582809 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:52.582820 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:52.582889 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:52.583027 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:52.583028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:52.583194 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:52.583240 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:52.583241 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:52.583877 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:52.583878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:52.584435 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:52.584438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:52.584762 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:52.584763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:52.584814 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:52.585304 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:52.585308 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:52.587121 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":80,"threads_started":1}
W20260619 08:56:52.587272 9598 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:52.582744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55593 (local address 127.0.98.62:41709)
0619 08:56:52.582816 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:52.582818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:52.582870 (+ 52us) server_negotiation.cc:407] Connection header received
0619 08:56:52.582953 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:52.582954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:52.582966 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:52.583014 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:52.583306 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:52.583804 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:52.584514 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:52.584705 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:52.584760 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:52.587209 (+ 2449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55593: BlockingRecv error: recv got EOF from 127.0.98.2:55593 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 19 08:56:52 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:52 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:52.710381 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:52.706054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.3:41317)
0619 08:56:52.706198 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:52.706208 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:52.706269 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:52.706378 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:52.706379 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:52.706554 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:52.706608 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:52.706611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:52.707399 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:52.707400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:52.707917 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:52.707919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:52.708278 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:52.708279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:52.708322 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:52.708723 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:52.708727 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:52.710322 (+ 1595us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
W20260619 08:56:52.710507 9598 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:52.706119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41317 (local address 127.0.98.62:41709)
0619 08:56:52.706172 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:52.706174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:52.706253 (+ 79us) server_negotiation.cc:407] Connection header received
0619 08:56:52.706309 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:52.706310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:52.706322 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:52.706369 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:52.706678 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:52.707327 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:52.707987 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:52.708227 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:52.708279 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:52.710412 (+ 2133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41317: BlockingRecv error: recv got EOF from 127.0.98.3:41317 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Jun 19 08:56:53 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:53 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:53.477813 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:53.473331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.1:45505)
0619 08:56:53.473451 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:53.473461 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:53.473506 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:53.473633 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:53.473634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:53.473784 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:53.473833 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:53.473834 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:53.474600 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:53.474601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:53.475135 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:53.475136 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:53.475544 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:53.475545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:53.475620 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:53.476135 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:53.476140 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:53.477742 (+ 1602us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
W20260619 08:56:53.477893 9602 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:53.473391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45505 (local address 127.0.98.62:41709)
0619 08:56:53.473504 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:53.473506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:53.473513 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:56:53.473560 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:53.473561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:53.473570 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:53.473618 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:53.473930 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:53.474530 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:53.475213 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:53.475446 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:53.475507 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:53.477830 (+ 2323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45505: BlockingRecv error: recv got EOF from 127.0.98.1:45505 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":57,"threads_started":1}
Jun 19 08:56:53 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:53 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:53.591859 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:53.587664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.2:55615)
0619 08:56:53.587784 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:53.587793 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:53.587843 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:53.587969 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:53.587970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:53.588154 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:53.588201 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:53.588202 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:53.588812 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:53.588812 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:53.589331 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:53.589332 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:53.589648 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:53.589649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:53.589688 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:53.590122 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:53.590145 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:53.591791 (+ 1646us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":55,"threads_started":1}
W20260619 08:56:53.591933 9602 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:53.587768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55615 (local address 127.0.98.62:41709)
0619 08:56:53.587819 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:53.587820 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:53.587831 (+ 11us) server_negotiation.cc:407] Connection header received
0619 08:56:53.587897 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:53.587897 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:53.587906 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:53.587958 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:53.588261 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:53.588751 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:53.589386 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:53.589573 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:53.589625 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:53.591876 (+ 2251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55615: BlockingRecv error: recv got EOF from 127.0.98.2:55615 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Jun 19 08:56:53 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:53 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:53.714911 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:53.710824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.3:36567)
0619 08:56:53.710960 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:53.710971 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:53.711030 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:53.711149 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:53.711150 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:53.711304 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:53.711350 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:53.711352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:53.711926 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:53.711926 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:53.712476 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:53.712478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:53.712804 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:53.712805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:53.712850 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:53.713260 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:53.713264 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:53.714836 (+ 1572us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":63,"threads_started":1}
W20260619 08:56:53.714977 9602 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:53.710890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36567 (local address 127.0.98.62:41709)
0619 08:56:53.710944 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:53.710946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:53.711023 (+ 77us) server_negotiation.cc:407] Connection header received
0619 08:56:53.711081 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:53.711082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:53.711092 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:53.711139 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:53.711397 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:53.711872 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:53.712540 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:53.712734 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:53.712788 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:53.714918 (+ 2130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36567: BlockingRecv error: recv got EOF from 127.0.98.3:36567 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Jun 19 08:56:54 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:54 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:54.486127 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:54.478320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.1:54923)
0619 08:56:54.478456 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:54.478466 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:54.478531 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:54.478669 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:54.478670 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:54.478822 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:54.478873 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:54.478875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:54.479563 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:54.479565 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:54.480194 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:54.480197 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:54.480622 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:54.480623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:54.480671 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:54.484315 (+ 3644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:54.484320 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:54.486053 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
W20260619 08:56:54.486249 9606 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:54.478434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54923 (local address 127.0.98.62:41709)
0619 08:56:54.478537 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:54.478539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:54.478566 (+ 27us) server_negotiation.cc:407] Connection header received
0619 08:56:54.478590 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:54.478590 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:54.478601 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:54.478646 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:54.478928 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:54.479433 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:54.480287 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:54.480564 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:54.480616 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:54.486159 (+ 5543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54923: BlockingRecv error: recv got EOF from 127.0.98.1:54923 (error 108)
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":46,"threads_started":1}
Jun 19 08:56:54 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:54 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:54.596565 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:54.592315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.2:34567)
0619 08:56:54.592443 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:54.592454 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:54.592515 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:54.592643 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:54.592644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:54.592811 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:54.592858 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:54.592860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:54.593495 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:54.593496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:54.594016 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:54.594017 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:54.594332 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:54.594333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:54.594387 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:54.594844 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:54.594848 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:54.596504 (+ 1656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":54,"threads_started":1}
W20260619 08:56:54.596652 9606 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:54.592438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34567 (local address 127.0.98.62:41709)
0619 08:56:54.592500 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:54.592502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:54.592512 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:56:54.592571 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:54.592572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:54.592584 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:54.592630 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:54.592932 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:54.593420 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:54.594088 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:54.594273 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:54.594327 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:54.596595 (+ 2268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34567: BlockingRecv error: recv got EOF from 127.0.98.2:34567 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 19 08:56:54 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:54 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:54.719765 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:54.715395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.3:41125)
0619 08:56:54.715551 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:54.715562 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:54.715633 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:54.715766 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:54.715767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:54.715941 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:54.715985 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:54.715987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:54.716701 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:54.716702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:54.717247 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:54.717249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:54.717565 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:54.717566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:54.717616 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:54.718045 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:54.718049 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:54.719708 (+ 1659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
W20260619 08:56:54.719930 9606 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:54.715475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41125 (local address 127.0.98.62:41709)
0619 08:56:54.715555 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:54.715557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:54.715614 (+ 57us) server_negotiation.cc:407] Connection header received
0619 08:56:54.715683 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:54.715684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:54.715698 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:54.715749 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:54.716055 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:54.716627 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:54.717325 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:54.717508 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:54.717563 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:54.719863 (+ 2300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41125: BlockingRecv error: recv got EOF from 127.0.98.3:41125 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 08:56:55 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:55 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:55.490903 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:55.486643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.1:52893)
0619 08:56:55.486796 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:55.486807 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:55.486866 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:55.486999 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:55.487000 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:55.487158 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:55.487206 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:55.487207 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:55.487831 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:55.487833 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:55.488411 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:55.488413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:55.488717 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:55.488718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:55.488762 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:55.489182 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:55.489186 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:55.490839 (+ 1653us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":67,"threads_started":1}
W20260619 08:56:55.491017 9610 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:55.486753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52893 (local address 127.0.98.62:41709)
0619 08:56:55.486865 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:55.486866 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:55.486875 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:56:55.486917 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:55.486917 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:55.486928 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:55.486984 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:55.487259 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:55.487744 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:55.488466 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:55.488648 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:55.488695 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:55.490947 (+ 2252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52893: BlockingRecv error: recv got EOF from 127.0.98.1:52893 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":53,"threads_started":1}
Jun 19 08:56:55 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:55 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:55.601431 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:55.597027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.2:56655)
0619 08:56:55.597148 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:55.597160 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:55.597229 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:55.597400 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:55.597400 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:55.597554 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:55.597618 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:55.597620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:55.598239 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:55.598240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:55.598859 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:55.598862 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:55.599276 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:55.599277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:55.599323 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:55.599709 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:55.599714 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:55.601360 (+ 1646us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":64,"threads_started":1}
W20260619 08:56:55.601503 9610 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:55.597098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56655 (local address 127.0.98.62:41709)
0619 08:56:55.597151 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:55.597154 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:55.597217 (+ 63us) server_negotiation.cc:407] Connection header received
0619 08:56:55.597305 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:55.597307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:55.597320 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:55.597388 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:55.597670 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:55.598161 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:55.598928 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:55.599209 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:55.599269 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:55.601445 (+ 2176us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56655: BlockingRecv error: recv got EOF from 127.0.98.2:56655 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Jun 19 08:56:55 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:55 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:55.724423 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:55.720214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.3:36997)
0619 08:56:55.720342 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:55.720353 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:55.720422 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:55.720538 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:55.720538 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:55.720710 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:55.720755 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:55.720757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:55.721432 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:55.721433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:55.721982 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:55.721984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:55.722301 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:55.722302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:55.722354 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:55.722748 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:55.722752 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:55.724351 (+ 1599us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
W20260619 08:56:55.724494 9610 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:55.720275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36997 (local address 127.0.98.62:41709)
0619 08:56:55.720339 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:55.720341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:55.720401 (+ 60us) server_negotiation.cc:407] Connection header received
0619 08:56:55.720470 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:55.720470 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:55.720483 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:55.720528 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:55.720822 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:55.721369 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:55.722061 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:55.722244 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:55.722298 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:55.724443 (+ 2145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36997: BlockingRecv error: recv got EOF from 127.0.98.3:36997 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Jun 19 08:56:56 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:56 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:56.495983 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:56.491470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.1:59419)
0619 08:56:56.491598 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:56.491608 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:56.491659 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:56.491853 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:56.491854 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:56.492035 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:56.492137 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:56.492139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:56.492785 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:56.492786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:56.493340 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:56.493341 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:56.493675 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:56.493676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:56.493806 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:56.494273 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:56.494277 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:56.495890 (+ 1613us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
W20260619 08:56:56.496034 9614 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:56.491571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59419 (local address 127.0.98.62:41709)
0619 08:56:56.491716 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:56.491718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:56.491727 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:56:56.491750 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:56.491751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:56.491761 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:56.491827 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:56.492214 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:56.492726 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:56.493397 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:56.493584 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:56.493628 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:56.495976 (+ 2348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59419: BlockingRecv error: recv got EOF from 127.0.98.1:59419 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":61,"threads_started":1}
Jun 19 08:56:56 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 08:56:56 dist-test-slave-h4g5 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 08:56:56.606127 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:56.601908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.2:42625)
0619 08:56:56.602037 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:56.602049 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:56.602106 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:56.602236 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:56.602237 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:56.602409 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:56.602455 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:56.602457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:56.603079 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:56.603080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:56.603605 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:56.603607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:56.603933 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:56.603934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:56.603989 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:56.604470 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:56.604474 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:56.606040 (+ 1566us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41709: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":56,"threads_started":1}
W20260619 08:56:56.606190 9614 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:56.602015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42625 (local address 127.0.98.62:41709)
0619 08:56:56.602078 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:56.602080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:56.602089 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:56:56.602167 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:56.602168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:56.602179 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:56.602222 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:56.602524 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:56.603008 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:56.603681 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:56.603875 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:56.603930 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:56.606138 (+ 2208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42625: BlockingRecv error: recv got EOF from 127.0.98.2:42625 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 08:56:56.654446 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 9139
I20260619 08:56:56.658929 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 9290
I20260619 08:56:56.662770 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 9422
I20260619 08:56:56.666241 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 9068
2026-06-19T08:56:56Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17320 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T08:56:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:56:56Z Disabled control of system clock
I20260619 08:56:56.691116 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44771
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44771
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 08:56:56.770107 9619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:56.770313 9619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:56.770330 9619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:56.771464 9619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:56:56.771497 9619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:56.771507 9619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:56.771518 9619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:56:56.771529 9619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:56:56.772890 9619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44771
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:56.773092 9619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:56.773315 9619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 08:56:56.777801 9619 server_base.cc:1034] running on GCE node
W20260619 08:56:56.777750 9624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:56.777875 9625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:56.777904 9627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:56.778352 9619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:56.778640 9619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:56.779778 9619 hybrid_clock.cc:648] HybridClock initialized: now 1781859416779761 us; error 34 us; skew 500 ppm
I20260619 08:56:56.781033 9619 webserver.cc:460] Webserver started at http://127.0.98.62:39271/ using document root <none> and password file <none>
I20260619 08:56:56.781243 9619 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:56.781291 9619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:56.781409 9619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:56.782413 9619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "da2fa6e0051746babb14ea13cbff6c28"
format_stamp: "Formatted at 2026-06-19 08:56:56 on dist-test-slave-h4g5"
server_key: "8462229be7891735cab0162caa2b9e5b"
server_key_iv: "10670243cadff50faf3841b93d97b676"
server_key_version: "encryptionkey@0"
I20260619 08:56:56.782739 9619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "da2fa6e0051746babb14ea13cbff6c28"
format_stamp: "Formatted at 2026-06-19 08:56:56 on dist-test-slave-h4g5"
server_key: "8462229be7891735cab0162caa2b9e5b"
server_key_iv: "10670243cadff50faf3841b93d97b676"
server_key_version: "encryptionkey@0"
I20260619 08:56:56.783897 9619 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:56:56.784641 9633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:56.784788 9619 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:56.784868 9619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "da2fa6e0051746babb14ea13cbff6c28"
format_stamp: "Formatted at 2026-06-19 08:56:56 on dist-test-slave-h4g5"
server_key: "8462229be7891735cab0162caa2b9e5b"
server_key_iv: "10670243cadff50faf3841b93d97b676"
server_key_version: "encryptionkey@0"
I20260619 08:56:56.784966 9619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:56.793327 9619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:56.793605 9619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:56.793732 9619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:56.797458 9619 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44771
I20260619 08:56:56.797487 9685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44771 every 8 connection(s)
I20260619 08:56:56.797796 9619 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:56:56.798403 9686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:56.801048 9686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da2fa6e0051746babb14ea13cbff6c28: Bootstrap starting.
I20260619 08:56:56.801585 9686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da2fa6e0051746babb14ea13cbff6c28: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:56.801815 9686 log.cc:824] T 00000000000000000000000000000000 P da2fa6e0051746babb14ea13cbff6c28: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:56.802420 9686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da2fa6e0051746babb14ea13cbff6c28: No bootstrap required, opened a new log
I20260619 08:56:56.803903 9686 raft_consensus.cc:348] T 00000000000000000000000000000000 P da2fa6e0051746babb14ea13cbff6c28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da2fa6e0051746babb14ea13cbff6c28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } }
I20260619 08:56:56.804039 9686 raft_consensus.cc:374] T 00000000000000000000000000000000 P da2fa6e0051746babb14ea13cbff6c28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:56.804059 9686 raft_consensus.cc:729] T 00000000000000000000000000000000 P da2fa6e0051746babb14ea13cbff6c28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da2fa6e0051746babb14ea13cbff6c28, State: Initialized, Role: FOLLOWER
I20260619 08:56:56.804203 9686 consensus_queue.cc:260] T 00000000000000000000000000000000 P da2fa6e0051746babb14ea13cbff6c28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da2fa6e0051746babb14ea13cbff6c28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } }
I20260619 08:56:56.804260 9686 raft_consensus.cc:388] T 00000000000000000000000000000000 P da2fa6e0051746babb14ea13cbff6c28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:56:56.804284 9686 raft_consensus.cc:482] T 00000000000000000000000000000000 P da2fa6e0051746babb14ea13cbff6c28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:56:56.804309 9686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da2fa6e0051746babb14ea13cbff6c28 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:56.804795 9686 raft_consensus.cc:504] T 00000000000000000000000000000000 P da2fa6e0051746babb14ea13cbff6c28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da2fa6e0051746babb14ea13cbff6c28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } }
I20260619 08:56:56.804872 9686 leader_election.cc:302] T 00000000000000000000000000000000 P da2fa6e0051746babb14ea13cbff6c28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da2fa6e0051746babb14ea13cbff6c28; no voters:
I20260619 08:56:56.804971 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 9619
I20260619 08:56:56.805009 9686 leader_election.cc:288] T 00000000000000000000000000000000 P da2fa6e0051746babb14ea13cbff6c28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:56:56.805058 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 08:56:56.805063 9689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da2fa6e0051746babb14ea13cbff6c28 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:56.805190 9689 raft_consensus.cc:686] T 00000000000000000000000000000000 P da2fa6e0051746babb14ea13cbff6c28 [term 1 LEADER]: Becoming Leader. State: Replica: da2fa6e0051746babb14ea13cbff6c28, State: Running, Role: LEADER
I20260619 08:56:56.805251 9686 sys_catalog.cc:564] T 00000000000000000000000000000000 P da2fa6e0051746babb14ea13cbff6c28 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:56:56.805267 392 external_mini_cluster.cc:1428] Setting key ae4808b1cda33d1fe09a3c068001b471
I20260619 08:56:56.805334 9689 consensus_queue.cc:237] T 00000000000000000000000000000000 P da2fa6e0051746babb14ea13cbff6c28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da2fa6e0051746babb14ea13cbff6c28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } }
I20260619 08:56:56.805882 9690 sys_catalog.cc:455] T 00000000000000000000000000000000 P da2fa6e0051746babb14ea13cbff6c28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da2fa6e0051746babb14ea13cbff6c28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da2fa6e0051746babb14ea13cbff6c28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } }
I20260619 08:56:56.805953 9690 sys_catalog.cc:458] T 00000000000000000000000000000000 P da2fa6e0051746babb14ea13cbff6c28 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:56.806264 9691 sys_catalog.cc:455] T 00000000000000000000000000000000 P da2fa6e0051746babb14ea13cbff6c28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da2fa6e0051746babb14ea13cbff6c28. Latest consensus state: current_term: 1 leader_uuid: "da2fa6e0051746babb14ea13cbff6c28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da2fa6e0051746babb14ea13cbff6c28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44771 } } }
I20260619 08:56:56.806335 9691 sys_catalog.cc:458] T 00000000000000000000000000000000 P da2fa6e0051746babb14ea13cbff6c28 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:56.806535 9695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:56:56.807042 9695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:56:56.808527 9695 catalog_manager.cc:1347] Generated new cluster ID: dda415eb43294e90adfbd2631b381785
I20260619 08:56:56.808580 9695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:56:56.810771 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:56.805716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51112 (local address 127.0.98.62:44771)
0619 08:56:56.806023 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:56.806032 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:56.806061 (+ 29us) server_negotiation.cc:407] Connection header received
0619 08:56:56.806175 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:56.806183 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:56.806253 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:56.806587 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:56.806680 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:56.808058 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:56.808936 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:56.809349 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:56.809457 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:56.810122 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:56.810126 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:56.810126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:56:56.810329 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:56.810361 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:56:56.810372 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:56.810427 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:56.810506 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 08:56:56.810584 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":70,"threads_started":1}
I20260619 08:56:56.815647 9695 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:56:56.816341 9695 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:56:56.822182 9695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da2fa6e0051746babb14ea13cbff6c28: Generated new TSK 0
I20260619 08:56:56.822357 9695 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:56:56.823369 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44771
--builtin_ntp_servers=127.0.98.20:34989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 08:56:56.901566 9710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:56.901753 9710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:56.901782 9710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:56.902889 9710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:56.902925 9710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:56.902956 9710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 08:56:56.904274 9710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:56.904497 9710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:56.904752 9710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:56.909667 9718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:56.909688 9715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:56.909695 9716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:56.909979 9710 server_base.cc:1034] running on GCE node
I20260619 08:56:56.910244 9710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:56.910454 9710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:56.911598 9710 hybrid_clock.cc:648] HybridClock initialized: now 1781859416911588 us; error 25 us; skew 500 ppm
I20260619 08:56:56.912784 9710 webserver.cc:460] Webserver started at http://127.0.98.1:38441/ using document root <none> and password file <none>
I20260619 08:56:56.913025 9710 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:56.913069 9710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:56.913188 9710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:56.914203 9710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-0/data/instance:
uuid: "624fc86f772e40f293e6a3853583b563"
format_stamp: "Formatted at 2026-06-19 08:56:56 on dist-test-slave-h4g5"
server_key: "06589a806b5f2c9e8d34cd7ba2d8909d"
server_key_iv: "e878aee6a16ff767a0f350258e185840"
server_key_version: "encryptionkey@0"
I20260619 08:56:56.914515 9710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-0/wal/instance:
uuid: "624fc86f772e40f293e6a3853583b563"
format_stamp: "Formatted at 2026-06-19 08:56:56 on dist-test-slave-h4g5"
server_key: "06589a806b5f2c9e8d34cd7ba2d8909d"
server_key_iv: "e878aee6a16ff767a0f350258e185840"
server_key_version: "encryptionkey@0"
I20260619 08:56:56.915704 9710 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:56:56.916430 9724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:56.916595 9710 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:56.916668 9710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-0/wal
uuid: "624fc86f772e40f293e6a3853583b563"
format_stamp: "Formatted at 2026-06-19 08:56:56 on dist-test-slave-h4g5"
server_key: "06589a806b5f2c9e8d34cd7ba2d8909d"
server_key_iv: "e878aee6a16ff767a0f350258e185840"
server_key_version: "encryptionkey@0"
I20260619 08:56:56.916736 9710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:56.928375 9710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:56.928663 9710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:56.928781 9710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:56.928974 9710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:56.929286 9710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:56.929333 9710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:56.929368 9710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:56.929391 9710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:56.935952 9710 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40809
I20260619 08:56:56.935987 9837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40809 every 8 connection(s)
I20260619 08:56:56.936327 9710 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
I20260619 08:56:56.937846 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 9710
I20260619 08:56:56.937930 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-0/wal/instance
I20260619 08:56:56.938117 392 external_mini_cluster.cc:1428] Setting key 2c72b0aa417506b4a71ee75188f2bab7
I20260619 08:56:56.939047 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44771
--builtin_ntp_servers=127.0.98.20:34989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 08:56:56.941923 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:56.937018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54425 (local address 127.0.98.62:44771)
0619 08:56:56.937091 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:56.937093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:56.937313 (+ 220us) server_negotiation.cc:407] Connection header received
0619 08:56:56.937855 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:56.937856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:56.937867 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:56.937915 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:56.938705 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:56.939309 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:56.940619 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:56.940813 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:56.940868 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:56.941659 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:56.941660 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:56.941660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:56:56.941664 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:56.941680 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:56:56.941684 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:56.941730 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:56.941809 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 08:56:56.941837 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 08:56:56.942126 9840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:56.937003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.98.1:54425)
0619 08:56:56.937195 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:56.937229 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:56.937734 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:56.937990 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:56.937995 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:56.938005 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:56.938607 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:56.938613 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:56.939395 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:56.939396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:56.940541 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:56.940543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:56.940885 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:56.940886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:56.941008 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:56.941498 (+ 490us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:56:56.941499 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:56.941537 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:56:56.941564 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:56.941740 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:56.941744 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:56.941747 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:56:56.941806 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 08:56:56.941905 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":92,"threads_started":1}
I20260619 08:56:56.942636 9838 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44771
I20260619 08:56:56.942721 9838 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:56.942896 9838 heartbeater.cc:511] Master 127.0.98.62:44771 requested a full tablet report, sending...
I20260619 08:56:56.943341 9650 ts_manager.cc:194] Registered new tserver with Master: 624fc86f772e40f293e6a3853583b563 (127.0.98.1:40809)
I20260619 08:56:56.943888 9650 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:54425
W20260619 08:56:57.018234 9841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:57.018443 9841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:57.018471 9841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:57.019650 9841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:57.019692 9841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:57.019726 9841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 08:56:57.021166 9841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:57.021417 9841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:57.021716 9841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:57.026511 9846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:57.026528 9849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:57.026719 9841 server_base.cc:1034] running on GCE node
W20260619 08:56:57.026503 9847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:57.026955 9841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:57.027194 9841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:57.028330 9841 hybrid_clock.cc:648] HybridClock initialized: now 1781859417028307 us; error 34 us; skew 500 ppm
I20260619 08:56:57.029703 9841 webserver.cc:460] Webserver started at http://127.0.98.2:39945/ using document root <none> and password file <none>
I20260619 08:56:57.029932 9841 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:57.029980 9841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:57.030129 9841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:57.031152 9841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-1/data/instance:
uuid: "aa0f10d2c9864c63889ad8a5f686a52e"
format_stamp: "Formatted at 2026-06-19 08:56:57 on dist-test-slave-h4g5"
server_key: "35d3cc8e5237ed8c4e084ee6ecb6061f"
server_key_iv: "d78fa91dfcf48af763bec11baefcc795"
server_key_version: "encryptionkey@0"
I20260619 08:56:57.031472 9841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-1/wal/instance:
uuid: "aa0f10d2c9864c63889ad8a5f686a52e"
format_stamp: "Formatted at 2026-06-19 08:56:57 on dist-test-slave-h4g5"
server_key: "35d3cc8e5237ed8c4e084ee6ecb6061f"
server_key_iv: "d78fa91dfcf48af763bec11baefcc795"
server_key_version: "encryptionkey@0"
I20260619 08:56:57.032678 9841 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:56:57.033322 9855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:57.033468 9841 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:57.033545 9841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-1/wal
uuid: "aa0f10d2c9864c63889ad8a5f686a52e"
format_stamp: "Formatted at 2026-06-19 08:56:57 on dist-test-slave-h4g5"
server_key: "35d3cc8e5237ed8c4e084ee6ecb6061f"
server_key_iv: "d78fa91dfcf48af763bec11baefcc795"
server_key_version: "encryptionkey@0"
I20260619 08:56:57.033601 9841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:57.049917 9841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:57.050207 9841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:57.050338 9841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:57.050551 9841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:57.050856 9841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:57.050889 9841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:57.050917 9841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:57.050933 9841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:57.056186 9841 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38955
I20260619 08:56:57.056248 9968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38955 every 8 connection(s)
I20260619 08:56:57.056526 9841 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
I20260619 08:56:57.061362 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:57.057187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36019 (local address 127.0.98.62:44771)
0619 08:56:57.057245 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:57.057246 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:57.057458 (+ 212us) server_negotiation.cc:407] Connection header received
0619 08:56:57.058016 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:57.058017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:57.058029 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:57.058066 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:57.058677 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:57.059157 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:57.060148 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:57.060318 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:57.060360 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:57.061142 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:57.061143 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:57.061143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:56:57.061148 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:57.061162 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:56:57.061167 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:57.061216 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:57.061271 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 08:56:57.061297 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 08:56:57.061533 9971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:57.057171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.98.2:36019)
0619 08:56:57.057341 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:57.057381 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:57.057913 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:57.058130 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:57.058133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:57.058141 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:57.058617 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:57.058621 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:57.059237 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:57.059238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:57.060065 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:57.060068 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:57.060371 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:57.060371 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:57.060473 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:57.060984 (+ 511us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:56:57.060984 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:57.061021 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:56:57.061044 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:57.061216 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:57.061218 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:57.061220 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:56:57.061264 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 08:56:57.061341 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":73,"threads_started":1}
I20260619 08:56:57.061960 9969 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44771
I20260619 08:56:57.062042 9969 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:57.062184 9969 heartbeater.cc:511] Master 127.0.98.62:44771 requested a full tablet report, sending...
I20260619 08:56:57.062487 9650 ts_manager.cc:194] Registered new tserver with Master: aa0f10d2c9864c63889ad8a5f686a52e (127.0.98.2:38955)
I20260619 08:56:57.062812 9650 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:36019
I20260619 08:56:57.064247 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 9841
I20260619 08:56:57.064313 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-1/wal/instance
I20260619 08:56:57.064446 392 external_mini_cluster.cc:1428] Setting key 1ff9e6a4781dc7a6642264ccc69c2c35
I20260619 08:56:57.065258 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44771
--builtin_ntp_servers=127.0.98.20:34989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 08:56:57.146381 9972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:57.146620 9972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:57.146641 9972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:57.147907 9972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:57.147958 9972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:57.147985 9972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 08:56:57.149446 9972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:57.149683 9972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:57.149976 9972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:57.154752 9977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:57.154763 9978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:57.154763 9980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:57.154824 9972 server_base.cc:1034] running on GCE node
I20260619 08:56:57.155177 9972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:57.155372 9972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:57.156497 9972 hybrid_clock.cc:648] HybridClock initialized: now 1781859417156481 us; error 35 us; skew 500 ppm
I20260619 08:56:57.157641 9972 webserver.cc:460] Webserver started at http://127.0.98.3:33929/ using document root <none> and password file <none>
I20260619 08:56:57.157835 9972 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:57.157876 9972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:57.157966 9972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:57.158964 9972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-2/data/instance:
uuid: "fe788a2dec5342559c8e0b5063cdfcd0"
format_stamp: "Formatted at 2026-06-19 08:56:57 on dist-test-slave-h4g5"
server_key: "ea0dc3548959cd6471c7c66e6ccea9e1"
server_key_iv: "10910a9c103fe23f271ea12f6f534e5f"
server_key_version: "encryptionkey@0"
I20260619 08:56:57.159283 9972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fe788a2dec5342559c8e0b5063cdfcd0"
format_stamp: "Formatted at 2026-06-19 08:56:57 on dist-test-slave-h4g5"
server_key: "ea0dc3548959cd6471c7c66e6ccea9e1"
server_key_iv: "10910a9c103fe23f271ea12f6f534e5f"
server_key_version: "encryptionkey@0"
I20260619 08:56:57.160434 9972 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:56:57.161054 9986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:57.161193 9972 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:57.161267 9972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-2/wal
uuid: "fe788a2dec5342559c8e0b5063cdfcd0"
format_stamp: "Formatted at 2026-06-19 08:56:57 on dist-test-slave-h4g5"
server_key: "ea0dc3548959cd6471c7c66e6ccea9e1"
server_key_iv: "10910a9c103fe23f271ea12f6f534e5f"
server_key_version: "encryptionkey@0"
I20260619 08:56:57.161330 9972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:57.174599 9972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:57.174901 9972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:57.175007 9972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:57.175197 9972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:57.175462 9972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:57.175495 9972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:57.175516 9972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:57.175542 9972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:57.180899 9972 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41673
I20260619 08:56:57.180974 10099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41673 every 8 connection(s)
I20260619 08:56:57.181264 9972 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
I20260619 08:56:57.186242 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:57.181931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59649 (local address 127.0.98.62:44771)
0619 08:56:57.181992 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:57.181993 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:57.182250 (+ 257us) server_negotiation.cc:407] Connection header received
0619 08:56:57.182745 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:57.182747 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:57.182757 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:57.182811 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:57.183440 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:57.183928 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:57.184989 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:57.185188 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:57.185241 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:57.186014 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:57.186015 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:57.186015 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:56:57.186033 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:57.186048 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:56:57.186051 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:57.186090 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:57.186152 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 08:56:57.186178 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 08:56:57.186424 10102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:57.181925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44771 (local address 127.0.98.3:59649)
0619 08:56:57.182108 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:57.182142 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:57.182641 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:57.182877 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:57.182881 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:57.182889 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:57.183368 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:57.183372 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:57.183998 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:57.183998 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:57.184913 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:57.184917 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:57.185252 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:57.185252 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:57.185367 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:57.185883 (+ 516us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:56:57.185884 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:57.185915 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:56:57.185941 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:57.186096 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:57.186099 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:57.186101 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:56:57.186149 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 08:56:57.186231 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":83,"threads_started":1}
I20260619 08:56:57.186856 10100 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44771
I20260619 08:56:57.186947 10100 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:57.187134 10100 heartbeater.cc:511] Master 127.0.98.62:44771 requested a full tablet report, sending...
I20260619 08:56:57.187458 9650 ts_manager.cc:194] Registered new tserver with Master: fe788a2dec5342559c8e0b5063cdfcd0 (127.0.98.3:41673)
I20260619 08:56:57.187847 9650 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:59649
I20260619 08:56:57.189769 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 9972
I20260619 08:56:57.189864 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859361540516-392-0/minicluster-data/ts-2/wal/instance
I20260619 08:56:57.190061 392 external_mini_cluster.cc:1428] Setting key c027e97ea373e74e5bedec4446e483cb
I20260619 08:56:57.190838 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 08:56:57.192490 9694 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:57.192113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51122 (local address 127.0.98.62:44771)
0619 08:56:57.192152 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:57.192154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:57.192217 (+ 63us) server_negotiation.cc:407] Connection header received
0619 08:56:57.192268 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:57.192268 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:57.192278 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:57.192316 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:57.192432 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51122: BlockingRecv error: recv got EOF from 127.0.0.1:51122 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 08:56:57.192559 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 9710
I20260619 08:56:57.197255 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 9841
I20260619 08:56:57.200922 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 9972
I20260619 08:56:57.204421 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 9619
2026-06-19T08:56:57Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (536 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T08:56:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:56:57Z Disabled control of system clock
I20260619 08:56:57.228475 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38397
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 08:56:57.305989 10111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:57.306210 10111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:57.306232 10111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:57.307358 10111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:56:57.307390 10111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:57.307401 10111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:57.307413 10111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:56:57.307427 10111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:56:57.308740 10111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38397
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:57.308939 10111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:57.309176 10111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:57.313789 10119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:57.313794 10117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:57.313978 10116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:57.314272 10111 server_base.cc:1034] running on GCE node
I20260619 08:56:57.314522 10111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:57.314838 10111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:57.315999 10111 hybrid_clock.cc:648] HybridClock initialized: now 1781859417315971 us; error 41 us; skew 500 ppm
I20260619 08:56:57.317471 10111 webserver.cc:460] Webserver started at http://127.0.98.62:35645/ using document root <none> and password file <none>
I20260619 08:56:57.317765 10111 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:57.317833 10111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:57.317967 10111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:57.319415 10111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "aaa606a513ba4ba4bbe78546c64280f7"
format_stamp: "Formatted at 2026-06-19 08:56:57 on dist-test-slave-h4g5"
server_key: "87e77652c0c864e701a867628d97aac3"
server_key_iv: "1a1067fb036dc3a495bedd30c157ee96"
server_key_version: "encryptionkey@0"
I20260619 08:56:57.319778 10111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "aaa606a513ba4ba4bbe78546c64280f7"
format_stamp: "Formatted at 2026-06-19 08:56:57 on dist-test-slave-h4g5"
server_key: "87e77652c0c864e701a867628d97aac3"
server_key_iv: "1a1067fb036dc3a495bedd30c157ee96"
server_key_version: "encryptionkey@0"
I20260619 08:56:57.321468 10111 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260619 08:56:57.322237 10125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:57.322387 10111 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 08:56:57.322480 10111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "aaa606a513ba4ba4bbe78546c64280f7"
format_stamp: "Formatted at 2026-06-19 08:56:57 on dist-test-slave-h4g5"
server_key: "87e77652c0c864e701a867628d97aac3"
server_key_iv: "1a1067fb036dc3a495bedd30c157ee96"
server_key_version: "encryptionkey@0"
I20260619 08:56:57.322542 10111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:57.325820 10111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:57.326085 10111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:57.326200 10111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:57.329975 10111 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38397
I20260619 08:56:57.330005 10177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38397 every 8 connection(s)
I20260619 08:56:57.330314 10111 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:56:57.330911 10178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:57.332032 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 10111
I20260619 08:56:57.332129 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 08:56:57.332302 392 external_mini_cluster.cc:1428] Setting key adcd5c78eae24ecd2b824d48a7bd80e9
I20260619 08:56:57.334291 10178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaa606a513ba4ba4bbe78546c64280f7: Bootstrap starting.
I20260619 08:56:57.334668 10181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:57.332794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56644 (local address 127.0.98.62:38397)
0619 08:56:57.332984 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:57.333001 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:57.333033 (+ 32us) server_negotiation.cc:407] Connection header received
0619 08:56:57.333519 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:57.333529 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:57.333585 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:57.333727 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:57.334250 (+ 523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:57.334253 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:57.334254 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:56:57.334258 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:57.334300 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:56:57.334314 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:57.334380 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:57.334485 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 08:56:57.334533 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
I20260619 08:56:57.335003 10178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aaa606a513ba4ba4bbe78546c64280f7: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:57.335268 10178 log.cc:824] T 00000000000000000000000000000000 P aaa606a513ba4ba4bbe78546c64280f7: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:57.335876 10178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaa606a513ba4ba4bbe78546c64280f7: No bootstrap required, opened a new log
I20260619 08:56:57.337193 10178 raft_consensus.cc:348] T 00000000000000000000000000000000 P aaa606a513ba4ba4bbe78546c64280f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaa606a513ba4ba4bbe78546c64280f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38397 } }
I20260619 08:56:57.337322 10178 raft_consensus.cc:374] T 00000000000000000000000000000000 P aaa606a513ba4ba4bbe78546c64280f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:57.337344 10178 raft_consensus.cc:729] T 00000000000000000000000000000000 P aaa606a513ba4ba4bbe78546c64280f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaa606a513ba4ba4bbe78546c64280f7, State: Initialized, Role: FOLLOWER
I20260619 08:56:57.337457 10178 consensus_queue.cc:260] T 00000000000000000000000000000000 P aaa606a513ba4ba4bbe78546c64280f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaa606a513ba4ba4bbe78546c64280f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38397 } }
I20260619 08:56:57.337548 10178 raft_consensus.cc:388] T 00000000000000000000000000000000 P aaa606a513ba4ba4bbe78546c64280f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:56:57.337579 10178 raft_consensus.cc:482] T 00000000000000000000000000000000 P aaa606a513ba4ba4bbe78546c64280f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:56:57.337625 10178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aaa606a513ba4ba4bbe78546c64280f7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:57.338114 10178 raft_consensus.cc:504] T 00000000000000000000000000000000 P aaa606a513ba4ba4bbe78546c64280f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaa606a513ba4ba4bbe78546c64280f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38397 } }
I20260619 08:56:57.338198 10178 leader_election.cc:302] T 00000000000000000000000000000000 P aaa606a513ba4ba4bbe78546c64280f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aaa606a513ba4ba4bbe78546c64280f7; no voters:
I20260619 08:56:57.338318 10178 leader_election.cc:288] T 00000000000000000000000000000000 P aaa606a513ba4ba4bbe78546c64280f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:56:57.338559 10178 sys_catalog.cc:564] T 00000000000000000000000000000000 P aaa606a513ba4ba4bbe78546c64280f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:56:57.338583 10183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aaa606a513ba4ba4bbe78546c64280f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:57.338702 10183 raft_consensus.cc:686] T 00000000000000000000000000000000 P aaa606a513ba4ba4bbe78546c64280f7 [term 1 LEADER]: Becoming Leader. State: Replica: aaa606a513ba4ba4bbe78546c64280f7, State: Running, Role: LEADER
I20260619 08:56:57.338799 10183 consensus_queue.cc:237] T 00000000000000000000000000000000 P aaa606a513ba4ba4bbe78546c64280f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaa606a513ba4ba4bbe78546c64280f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38397 } }
I20260619 08:56:57.339099 10184 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaa606a513ba4ba4bbe78546c64280f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aaa606a513ba4ba4bbe78546c64280f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaa606a513ba4ba4bbe78546c64280f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38397 } } }
I20260619 08:56:57.339178 10184 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaa606a513ba4ba4bbe78546c64280f7 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:57.339332 10185 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaa606a513ba4ba4bbe78546c64280f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aaa606a513ba4ba4bbe78546c64280f7. Latest consensus state: current_term: 1 leader_uuid: "aaa606a513ba4ba4bbe78546c64280f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaa606a513ba4ba4bbe78546c64280f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38397 } } }
I20260619 08:56:57.339385 10185 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaa606a513ba4ba4bbe78546c64280f7 [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:57.339704 10192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:56:57.340196 10192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:56:57.341585 10192 catalog_manager.cc:1347] Generated new cluster ID: 44c26603f0ac44359a98c35d3b349ba7
I20260619 08:56:57.341627 10192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:56:57.354748 10192 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:56:57.354962 10192 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:56:57.367856 10192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aaa606a513ba4ba4bbe78546c64280f7: Generated new TSK 0
I20260619 08:56:57.368126 10192 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:56:57.377857 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38397
--builtin_ntp_servers=127.0.98.20:45085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 08:56:57.476251 10202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:57.476467 10202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:57.476490 10202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:57.477677 10202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:57.477730 10202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:57.477756 10202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 08:56:57.479202 10202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:57.479444 10202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:57.479729 10202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:57.484581 10207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:57.484581 10208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:57.484577 10210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:57.484882 10202 server_base.cc:1034] running on GCE node
I20260619 08:56:57.485052 10202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:57.485283 10202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:57.486413 10202 hybrid_clock.cc:648] HybridClock initialized: now 1781859417486399 us; error 27 us; skew 500 ppm
I20260619 08:56:57.487576 10202 webserver.cc:460] Webserver started at http://127.0.98.1:38357/ using document root <none> and password file <none>
I20260619 08:56:57.487802 10202 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:57.487847 10202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:57.487972 10202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:57.489084 10202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-0/data/instance:
uuid: "ebe7983533544141a480b37cb12cdf1b"
format_stamp: "Formatted at 2026-06-19 08:56:57 on dist-test-slave-h4g5"
server_key: "033e6a4b2b09b78d1cd78bbb5b259ee2"
server_key_iv: "a28ba5d2fe7321453197be72fa06a908"
server_key_version: "encryptionkey@0"
I20260619 08:56:57.489492 10202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ebe7983533544141a480b37cb12cdf1b"
format_stamp: "Formatted at 2026-06-19 08:56:57 on dist-test-slave-h4g5"
server_key: "033e6a4b2b09b78d1cd78bbb5b259ee2"
server_key_iv: "a28ba5d2fe7321453197be72fa06a908"
server_key_version: "encryptionkey@0"
I20260619 08:56:57.490935 10202 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:56:57.491547 10216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:57.491689 10202 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:57.491770 10202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-0/wal
uuid: "ebe7983533544141a480b37cb12cdf1b"
format_stamp: "Formatted at 2026-06-19 08:56:57 on dist-test-slave-h4g5"
server_key: "033e6a4b2b09b78d1cd78bbb5b259ee2"
server_key_iv: "a28ba5d2fe7321453197be72fa06a908"
server_key_version: "encryptionkey@0"
I20260619 08:56:57.491839 10202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:57.495190 10202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:57.495440 10202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:57.495553 10202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:57.495743 10202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:57.496022 10202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:57.496054 10202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:57.496112 10202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:57.496138 10202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:57.501413 10202 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45315
I20260619 08:56:57.501464 10329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45315 every 8 connection(s)
I20260619 08:56:57.501751 10202 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
I20260619 08:56:57.503052 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 10202
I20260619 08:56:57.503134 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-0/wal/instance
I20260619 08:56:57.503327 392 external_mini_cluster.cc:1428] Setting key 2914406101239da736fda191710fb4c8
I20260619 08:56:57.504243 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38397
--builtin_ntp_servers=127.0.98.20:45085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 08:56:57.504752 10181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:57.502401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54149 (local address 127.0.98.62:38397)
0619 08:56:57.502451 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:57.502452 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:57.502645 (+ 193us) server_negotiation.cc:407] Connection header received
0619 08:56:57.503217 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:57.503218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:57.503223 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:57.503263 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:57.504527 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:57.504528 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:57.504529 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:56:57.504534 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:57.504552 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:56:57.504557 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:57.504622 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:57.504672 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 08:56:57.504685 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 08:56:57.504912 10332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:57.502380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38397 (local address 127.0.98.1:54149)
0619 08:56:57.502541 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:57.502574 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:57.503120 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:57.503341 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:57.503345 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:57.503354 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:57.504375 (+ 1021us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:56:57.504375 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:57.504414 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:56:57.504449 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:57.504622 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:57.504625 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:57.504628 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:56:57.504677 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 08:56:57.504723 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":82,"threads_started":1}
I20260619 08:56:57.505360 10330 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38397
I20260619 08:56:57.505460 10330 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:57.505592 10330 heartbeater.cc:511] Master 127.0.98.62:38397 requested a full tablet report, sending...
I20260619 08:56:57.506038 10142 ts_manager.cc:194] Registered new tserver with Master: ebe7983533544141a480b37cb12cdf1b (127.0.98.1:45315)
W20260619 08:56:57.583739 10333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:57.583961 10333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:57.583983 10333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:57.585161 10333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:57.585202 10333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:57.585228 10333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 08:56:57.586561 10333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:57.586781 10333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:57.587073 10333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:57.591826 10338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:57.591822 10341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:57.592031 10339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:57.592182 10333 server_base.cc:1034] running on GCE node
I20260619 08:56:57.592378 10333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:57.592630 10333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:57.593796 10333 hybrid_clock.cc:648] HybridClock initialized: now 1781859417593779 us; error 30 us; skew 500 ppm
I20260619 08:56:57.595100 10333 webserver.cc:460] Webserver started at http://127.0.98.2:38209/ using document root <none> and password file <none>
I20260619 08:56:57.595338 10333 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:57.595387 10333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:57.595546 10333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:57.596655 10333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-1/data/instance:
uuid: "647e440ecd7a4185a1ca0306687a188a"
format_stamp: "Formatted at 2026-06-19 08:56:57 on dist-test-slave-h4g5"
server_key: "e811825fd34cef213cdbcdc146104ec6"
server_key_iv: "23664595a82302b5f020c62324049921"
server_key_version: "encryptionkey@0"
I20260619 08:56:57.596973 10333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-1/wal/instance:
uuid: "647e440ecd7a4185a1ca0306687a188a"
format_stamp: "Formatted at 2026-06-19 08:56:57 on dist-test-slave-h4g5"
server_key: "e811825fd34cef213cdbcdc146104ec6"
server_key_iv: "23664595a82302b5f020c62324049921"
server_key_version: "encryptionkey@0"
I20260619 08:56:57.598202 10333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:56:57.598923 10347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:57.599050 10333 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:57.599126 10333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-1/wal
uuid: "647e440ecd7a4185a1ca0306687a188a"
format_stamp: "Formatted at 2026-06-19 08:56:57 on dist-test-slave-h4g5"
server_key: "e811825fd34cef213cdbcdc146104ec6"
server_key_iv: "23664595a82302b5f020c62324049921"
server_key_version: "encryptionkey@0"
I20260619 08:56:57.599195 10333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:57.602429 10333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:57.602669 10333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:57.602787 10333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:57.602988 10333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:57.603261 10333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:57.603291 10333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:57.603320 10333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:57.603343 10333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:57.608698 10333 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43523
I20260619 08:56:57.608767 10460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43523 every 8 connection(s)
I20260619 08:56:57.609036 10333 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
I20260619 08:56:57.611836 10181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:57.609707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38705 (local address 127.0.98.62:38397)
0619 08:56:57.609756 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:57.609758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:57.609961 (+ 203us) server_negotiation.cc:407] Connection header received
0619 08:56:57.610451 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:57.610452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:57.610460 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:57.610519 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:57.611600 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:57.611601 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:57.611601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:56:57.611606 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:57.611623 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:56:57.611627 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:57.611676 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:57.611750 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 08:56:57.611765 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 08:56:57.612010 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:57.609692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38397 (local address 127.0.98.2:38705)
0619 08:56:57.609846 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:57.609879 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:57.610352 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:57.610603 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:57.610607 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:57.610616 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:57.611447 (+ 831us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:56:57.611447 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:57.611475 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:56:57.611502 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:57.611679 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:57.611683 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:57.611687 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:56:57.611757 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 08:56:57.611818 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":63,"threads_started":1}
I20260619 08:56:57.612438 10461 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38397
I20260619 08:56:57.612517 10461 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:57.612645 10461 heartbeater.cc:511] Master 127.0.98.62:38397 requested a full tablet report, sending...
I20260619 08:56:57.612949 10142 ts_manager.cc:194] Registered new tserver with Master: 647e440ecd7a4185a1ca0306687a188a (127.0.98.2:43523)
I20260619 08:56:57.619163 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 10333
I20260619 08:56:57.619244 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-1/wal/instance
I20260619 08:56:57.619377 392 external_mini_cluster.cc:1428] Setting key c23ba875f966c50b16f1e7eb6c3a64ec
I20260619 08:56:57.620260 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38397
--builtin_ntp_servers=127.0.98.20:45085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 08:56:57.699313 10464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:57.699517 10464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:57.699538 10464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:57.700697 10464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:57.700736 10464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:57.700762 10464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 08:56:57.702076 10464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:57.702275 10464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:57.702509 10464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:57.707239 10469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:57.707247 10464 server_base.cc:1034] running on GCE node
W20260619 08:56:57.707247 10470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:57.707311 10472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:57.707619 10464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:57.707840 10464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:57.708976 10464 hybrid_clock.cc:648] HybridClock initialized: now 1781859417708959 us; error 34 us; skew 500 ppm
I20260619 08:56:57.710134 10464 webserver.cc:460] Webserver started at http://127.0.98.3:37489/ using document root <none> and password file <none>
I20260619 08:56:57.710347 10464 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:57.710394 10464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:57.710520 10464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:57.711512 10464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-2/data/instance:
uuid: "d371976e2b954f1f9c90b5b4095b6983"
format_stamp: "Formatted at 2026-06-19 08:56:57 on dist-test-slave-h4g5"
server_key: "76ac4f6523a01420c646fd0979a57f04"
server_key_iv: "51ef07e5dcee98378ce7dbea06338180"
server_key_version: "encryptionkey@0"
I20260619 08:56:57.711844 10464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d371976e2b954f1f9c90b5b4095b6983"
format_stamp: "Formatted at 2026-06-19 08:56:57 on dist-test-slave-h4g5"
server_key: "76ac4f6523a01420c646fd0979a57f04"
server_key_iv: "51ef07e5dcee98378ce7dbea06338180"
server_key_version: "encryptionkey@0"
I20260619 08:56:57.713078 10464 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:56:57.713725 10478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:57.713881 10464 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:57.713959 10464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-2/wal
uuid: "d371976e2b954f1f9c90b5b4095b6983"
format_stamp: "Formatted at 2026-06-19 08:56:57 on dist-test-slave-h4g5"
server_key: "76ac4f6523a01420c646fd0979a57f04"
server_key_iv: "51ef07e5dcee98378ce7dbea06338180"
server_key_version: "encryptionkey@0"
I20260619 08:56:57.714025 10464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:57.717260 10464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:57.717502 10464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:57.717624 10464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:57.717811 10464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:57.718102 10464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:57.718133 10464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:57.718158 10464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:57.718184 10464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:57.723465 10464 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33643
I20260619 08:56:57.723515 10591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33643 every 8 connection(s)
I20260619 08:56:57.723819 10464 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
I20260619 08:56:57.724519 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 10464
I20260619 08:56:57.724615 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859361540516-392-0/minicluster-data/ts-2/wal/instance
I20260619 08:56:57.724799 392 external_mini_cluster.cc:1428] Setting key 5c86654f098a3e0aec6cd723538f552e
I20260619 08:56:57.726727 10181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:57.724467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60121 (local address 127.0.98.62:38397)
0619 08:56:57.724513 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:57.724515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:57.724803 (+ 288us) server_negotiation.cc:407] Connection header received
0619 08:56:57.725348 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:57.725349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:57.725355 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:57.725418 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:57.726489 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:57.726490 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:57.726491 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:56:57.726495 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:57.726509 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:56:57.726513 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:57.726561 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:57.726643 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 08:56:57.726657 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 08:56:57.726900 10594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:57.724472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38397 (local address 127.0.98.3:60121)
0619 08:56:57.724663 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:57.724708 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:57.725262 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:57.725487 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:57.725491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:57.725501 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:57.726349 (+ 848us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:56:57.726350 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:57.726378 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:56:57.726406 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:57.726575 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:57.726580 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:57.726583 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:56:57.726644 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 08:56:57.726705 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":95,"threads_started":1}
I20260619 08:56:57.727305 10592 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38397
I20260619 08:56:57.727407 10592 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:57.727526 10592 heartbeater.cc:511] Master 127.0.98.62:38397 requested a full tablet report, sending...
I20260619 08:56:57.727880 10142 ts_manager.cc:194] Registered new tserver with Master: d371976e2b954f1f9c90b5b4095b6983 (127.0.98.3:33643)
I20260619 08:56:57.736549 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 08:56:57.738336 10181 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:56:57.737870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56646 (local address 127.0.98.62:38397)
0619 08:56:57.737944 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:57.737946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:57.738028 (+ 82us) server_negotiation.cc:407] Connection header received
0619 08:56:57.738102 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:57.738103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:57.738109 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:57.738150 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:57.738277 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56646: BlockingRecv error: recv got EOF from 127.0.0.1:56646 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 08:56:57.738588 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 10202
I20260619 08:56:57.742553 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 10333
I20260619 08:56:57.746349 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 10464
I20260619 08:56:57.749929 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 10111
2026-06-19T08:56:57Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (546 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:56:57 dist-test-slave-h4g5 krb5kdc[10602](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:56:57 dist-test-slave-h4g5 krb5kdc[10602](info): set up 2 sockets
Jun 19 08:56:57 dist-test-slave-h4g5 krb5kdc[10602](info): commencing operation
krb5kdc: starting...
W20260619 08:56:58.700675 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 08:56:58 dist-test-slave-h4g5 krb5kdc[10602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:56:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:56:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 08:56:58.812582 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44529
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44529
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:58.889917 10615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:58.890136 10615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:58.890165 10615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:58.891351 10615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:56:58.891395 10615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:58.891413 10615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:58.891428 10615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:56:58.891443 10615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:56:58.892964 10615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:58.893211 10615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:58.893422 10615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:58.897927 10621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:58.897923 10623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:58.897921 10620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:58.898216 10615 server_base.cc:1034] running on GCE node
I20260619 08:56:58.898487 10615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:58.898739 10615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:58.899869 10615 hybrid_clock.cc:648] HybridClock initialized: now 1781859418899855 us; error 28 us; skew 500 ppm
Jun 19 08:56:58 dist-test-slave-h4g5 krb5kdc[10602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:58.902207 10615 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 08:56:58.902606 10615 webserver.cc:460] Webserver started at http://127.0.98.62:38473/ using document root <none> and password file <none>
I20260619 08:56:58.902796 10615 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:58.902843 10615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:58.902961 10615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:58.903980 10615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "900887889557490fb9ab5308a07b83fc"
format_stamp: "Formatted at 2026-06-19 08:56:58 on dist-test-slave-h4g5"
server_key: "d0abacccf9f7499db694203e9c663c49"
server_key_iv: "4a585f80ea2720df4b4cd099a1acac85"
server_key_version: "encryptionkey@0"
I20260619 08:56:58.904356 10615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "900887889557490fb9ab5308a07b83fc"
format_stamp: "Formatted at 2026-06-19 08:56:58 on dist-test-slave-h4g5"
server_key: "d0abacccf9f7499db694203e9c663c49"
server_key_iv: "4a585f80ea2720df4b4cd099a1acac85"
server_key_version: "encryptionkey@0"
I20260619 08:56:58.905508 10615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:58.906139 10630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:58.906304 10615 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:58.906381 10615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "900887889557490fb9ab5308a07b83fc"
format_stamp: "Formatted at 2026-06-19 08:56:58 on dist-test-slave-h4g5"
server_key: "d0abacccf9f7499db694203e9c663c49"
server_key_iv: "4a585f80ea2720df4b4cd099a1acac85"
server_key_version: "encryptionkey@0"
I20260619 08:56:58.906446 10615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:58.919512 10615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:58.921298 10615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:58.921422 10615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:58.924981 10615 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44529
I20260619 08:56:58.925011 10682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44529 every 8 connection(s)
I20260619 08:56:58.925312 10615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:56:58.926215 10683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:58.926568 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 10615
I20260619 08:56:58.926642 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 08:56:58.926815 392 external_mini_cluster.cc:1428] Setting key fa8186e6d3dd63b79cbe0a14b64c1663
I20260619 08:56:58.929785 10683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 900887889557490fb9ab5308a07b83fc: Bootstrap starting.
I20260619 08:56:58.930411 10683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 900887889557490fb9ab5308a07b83fc: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:58.930727 10683 log.cc:824] T 00000000000000000000000000000000 P 900887889557490fb9ab5308a07b83fc: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:58.931380 10683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 900887889557490fb9ab5308a07b83fc: No bootstrap required, opened a new log
I20260619 08:56:58.933250 10683 raft_consensus.cc:348] T 00000000000000000000000000000000 P 900887889557490fb9ab5308a07b83fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "900887889557490fb9ab5308a07b83fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44529 } }
I20260619 08:56:58.933424 10683 raft_consensus.cc:374] T 00000000000000000000000000000000 P 900887889557490fb9ab5308a07b83fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:58.933458 10683 raft_consensus.cc:729] T 00000000000000000000000000000000 P 900887889557490fb9ab5308a07b83fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 900887889557490fb9ab5308a07b83fc, State: Initialized, Role: FOLLOWER
Jun 19 08:56:58 dist-test-slave-h4g5 krb5kdc[10602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:58.933583 10683 consensus_queue.cc:260] T 00000000000000000000000000000000 P 900887889557490fb9ab5308a07b83fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "900887889557490fb9ab5308a07b83fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44529 } }
I20260619 08:56:58.933676 10683 raft_consensus.cc:388] T 00000000000000000000000000000000 P 900887889557490fb9ab5308a07b83fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:56:58.933727 10683 raft_consensus.cc:482] T 00000000000000000000000000000000 P 900887889557490fb9ab5308a07b83fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:56:58.933774 10683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 900887889557490fb9ab5308a07b83fc [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:58.934268 10683 raft_consensus.cc:504] T 00000000000000000000000000000000 P 900887889557490fb9ab5308a07b83fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "900887889557490fb9ab5308a07b83fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44529 } }
I20260619 08:56:58.934352 10683 leader_election.cc:302] T 00000000000000000000000000000000 P 900887889557490fb9ab5308a07b83fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 900887889557490fb9ab5308a07b83fc; no voters:
I20260619 08:56:58.934499 10683 leader_election.cc:288] T 00000000000000000000000000000000 P 900887889557490fb9ab5308a07b83fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:56:58.934566 10688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 900887889557490fb9ab5308a07b83fc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:58.934725 10683 sys_catalog.cc:564] T 00000000000000000000000000000000 P 900887889557490fb9ab5308a07b83fc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:56:58.934711 10688 raft_consensus.cc:686] T 00000000000000000000000000000000 P 900887889557490fb9ab5308a07b83fc [term 1 LEADER]: Becoming Leader. State: Replica: 900887889557490fb9ab5308a07b83fc, State: Running, Role: LEADER
I20260619 08:56:58.934864 10688 consensus_queue.cc:237] T 00000000000000000000000000000000 P 900887889557490fb9ab5308a07b83fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "900887889557490fb9ab5308a07b83fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44529 } }
I20260619 08:56:58.935235 10690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 900887889557490fb9ab5308a07b83fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 900887889557490fb9ab5308a07b83fc. Latest consensus state: current_term: 1 leader_uuid: "900887889557490fb9ab5308a07b83fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "900887889557490fb9ab5308a07b83fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44529 } } }
I20260619 08:56:58.935320 10690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 900887889557490fb9ab5308a07b83fc [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:58.935458 10689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 900887889557490fb9ab5308a07b83fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "900887889557490fb9ab5308a07b83fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "900887889557490fb9ab5308a07b83fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44529 } } }
I20260619 08:56:58.935518 10689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 900887889557490fb9ab5308a07b83fc [sys.catalog]: This master's current role is: LEADER
I20260619 08:56:58.935580 10697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:56:58.936144 10697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:56:58.937849 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:58.927297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52048 (local address 127.0.98.62:44529)
0619 08:56:58.927486 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:58.927495 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:58.927527 (+ 32us) server_negotiation.cc:407] Connection header received
0619 08:56:58.928158 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:58.928167 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:58.928228 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:58.928338 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:58.929188 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:58.930119 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:58.931125 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:58.931562 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:58.931667 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:58.934151 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:58.934155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:58.934156 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:58.934168 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:58.936299 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:58.936628 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:58.936631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:58.936631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:58.936745 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:58.936961 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:58.936962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:58.936962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:58.937260 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:58.937401 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:58.937521 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 08:56:58.937658 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":83,"threads_started":1}
I20260619 08:56:58.941156 10697 catalog_manager.cc:1347] Generated new cluster ID: 0bd23a2849254a929161cf64cc14001e
I20260619 08:56:58.941226 10697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:56:58.948653 10697 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:56:58.949110 10697 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:56:58.953974 10697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 900887889557490fb9ab5308a07b83fc: Generated new TSK 0
I20260619 08:56:58.954120 10697 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 08:56:58.983919 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44529
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:59.061993 10709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:59.062186 10709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:59.062208 10709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:59.063273 10709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:59.063305 10709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:59.063328 10709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 08:56:59.064716 10709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:59.064890 10709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:59.065066 10709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:59.069689 10717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:59.069734 10715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:59.069746 10714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:59.069891 10709 server_base.cc:1034] running on GCE node
I20260619 08:56:59.070065 10709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:59.070297 10709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:59.071439 10709 hybrid_clock.cc:648] HybridClock initialized: now 1781859419071416 us; error 34 us; skew 500 ppm
Jun 19 08:56:59 dist-test-slave-h4g5 krb5kdc[10602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:59.073974 10709 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 08:56:59.074396 10709 webserver.cc:460] Webserver started at http://127.0.98.1:45949/ using document root <none> and password file <none>
I20260619 08:56:59.074594 10709 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:59.074651 10709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:59.074805 10709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:59.075825 10709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-0/data/instance:
uuid: "cc9c5351059541e7b28d896485e91b6e"
format_stamp: "Formatted at 2026-06-19 08:56:59 on dist-test-slave-h4g5"
server_key: "dc91a4e8cb981acd07302e050db49f50"
server_key_iv: "5c547e34ccd1d95d61f84e1877fd4bb3"
server_key_version: "encryptionkey@0"
I20260619 08:56:59.076179 10709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cc9c5351059541e7b28d896485e91b6e"
format_stamp: "Formatted at 2026-06-19 08:56:59 on dist-test-slave-h4g5"
server_key: "dc91a4e8cb981acd07302e050db49f50"
server_key_iv: "5c547e34ccd1d95d61f84e1877fd4bb3"
server_key_version: "encryptionkey@0"
I20260619 08:56:59.077394 10709 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:56:59.078092 10724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:59.078244 10709 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:59.078318 10709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-0/wal
uuid: "cc9c5351059541e7b28d896485e91b6e"
format_stamp: "Formatted at 2026-06-19 08:56:59 on dist-test-slave-h4g5"
server_key: "dc91a4e8cb981acd07302e050db49f50"
server_key_iv: "5c547e34ccd1d95d61f84e1877fd4bb3"
server_key_version: "encryptionkey@0"
I20260619 08:56:59.078387 10709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:59.117208 10709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:59.119396 10709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:59.119515 10709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:59.119696 10709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:59.119992 10709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:59.120028 10709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:59.120059 10709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:59.120105 10709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:59.125520 10709 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43685
I20260619 08:56:59.125579 10837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43685 every 8 connection(s)
I20260619 08:56:59.125873 10709 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
I20260619 08:56:59.128618 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 10709
I20260619 08:56:59.128685 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-0/wal/instance
I20260619 08:56:59.128861 392 external_mini_cluster.cc:1428] Setting key f6bb8ec2e1b230e72d1a042f279eb57a
Jun 19 08:56:59 dist-test-slave-h4g5 krb5kdc[10602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:59.135299 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:59.126560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52713 (local address 127.0.98.62:44529)
0619 08:56:59.126619 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:59.126621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:59.126796 (+ 175us) server_negotiation.cc:407] Connection header received
0619 08:56:59.127274 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:59.127275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:59.127284 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:59.127338 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:59.128272 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:59.128789 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:59.130087 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:59.130299 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:59.130350 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:59.132502 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:59.132504 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:59.132504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:59.132511 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:59.133870 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:59.134311 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:59.134312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:59.134313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:59.134357 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:59.134637 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:59.134638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:59.134638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:59.134751 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:59.134798 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:59.135152 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 08:56:59.135233 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 08:56:59.135460 10840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:59.126516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44529 (local address 127.0.98.1:52713)
0619 08:56:59.126676 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:59.126718 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:59.127176 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:59.127402 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:59.127405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:59.127954 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:59.128193 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:59.128198 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:59.128870 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:59.128871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:59.130013 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:59.130016 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:59.130369 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:59.130370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:59.130507 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:59.131028 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:59.131036 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:59.132420 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:59.133951 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:59.133958 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:59.133960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:59.134246 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:59.134414 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:59.134415 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:59.134415 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:59.134545 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:59.134797 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:59.134803 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:59.134962 (+ 159us) client_negotiation.cc:769] Sending connection context
0619 08:56:59.135079 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 08:56:59.135237 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
I20260619 08:56:59.135918 10838 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44529
I20260619 08:56:59.136003 10838 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:59.136243 10838 heartbeater.cc:511] Master 127.0.98.62:44529 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 08:56:59.136788 10647 ts_manager.cc:194] Registered new tserver with Master: cc9c5351059541e7b28d896485e91b6e (127.0.98.1:43685)
I20260619 08:56:59.137354 10647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52713
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 08:56:59.152151 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44529
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:59.230768 10843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:59.230995 10843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:59.231020 10843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:59.232218 10843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:59.232261 10843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:59.232286 10843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 08:56:59.233726 10843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:59.233913 10843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:59.234102 10843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:59.238852 10849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:59.238960 10843 server_base.cc:1034] running on GCE node
W20260619 08:56:59.238974 10851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:59.238895 10848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:59.239318 10843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:59.239517 10843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:59.240649 10843 hybrid_clock.cc:648] HybridClock initialized: now 1781859419240636 us; error 31 us; skew 500 ppm
Jun 19 08:56:59 dist-test-slave-h4g5 krb5kdc[10602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:59.242931 10843 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 08:56:59.243326 10843 webserver.cc:460] Webserver started at http://127.0.98.2:45427/ using document root <none> and password file <none>
I20260619 08:56:59.243520 10843 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:59.243566 10843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:59.243682 10843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:59.244771 10843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-1/data/instance:
uuid: "f097d806482549f8a65419d503ad6de9"
format_stamp: "Formatted at 2026-06-19 08:56:59 on dist-test-slave-h4g5"
server_key: "a5590dd328ca5cb707d63c8369fe7b04"
server_key_iv: "43e0e5ae2c445cb5076373742aedc000"
server_key_version: "encryptionkey@0"
I20260619 08:56:59.245105 10843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f097d806482549f8a65419d503ad6de9"
format_stamp: "Formatted at 2026-06-19 08:56:59 on dist-test-slave-h4g5"
server_key: "a5590dd328ca5cb707d63c8369fe7b04"
server_key_iv: "43e0e5ae2c445cb5076373742aedc000"
server_key_version: "encryptionkey@0"
I20260619 08:56:59.246301 10843 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:56:59.246941 10858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:59.247079 10843 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:59.247157 10843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-1/wal
uuid: "f097d806482549f8a65419d503ad6de9"
format_stamp: "Formatted at 2026-06-19 08:56:59 on dist-test-slave-h4g5"
server_key: "a5590dd328ca5cb707d63c8369fe7b04"
server_key_iv: "43e0e5ae2c445cb5076373742aedc000"
server_key_version: "encryptionkey@0"
I20260619 08:56:59.247215 10843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:59.273046 10843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:59.274822 10843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:59.274955 10843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:59.275163 10843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:59.275463 10843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:59.275501 10843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:59.275530 10843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:59.275547 10843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:59.280853 10843 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33443
I20260619 08:56:59.280912 10971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33443 every 8 connection(s)
I20260619 08:56:59.281248 10843 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
I20260619 08:56:59.286700 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 10843
I20260619 08:56:59.286791 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-1/wal/instance
Jun 19 08:56:59 dist-test-slave-h4g5 krb5kdc[10602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:59.286957 392 external_mini_cluster.cc:1428] Setting key 8f7327f902e0769d2dfc16a943d4512e
I20260619 08:56:59.289912 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:59.281901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53787 (local address 127.0.98.62:44529)
0619 08:56:59.281947 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:59.281949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:59.282121 (+ 172us) server_negotiation.cc:407] Connection header received
0619 08:56:59.282587 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:59.282589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:59.282600 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:59.282659 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:59.283577 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:59.284071 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:59.285079 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:59.285287 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:59.285352 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:59.287395 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:59.287397 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:59.287397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:59.287414 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:59.288644 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:59.289050 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:59.289060 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:59.289061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:59.289114 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:59.289324 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:59.289325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:59.289325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:59.289430 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:59.289484 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:59.289747 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 08:56:59.289826 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 08:56:59.290078 10974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:59.281838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44529 (local address 127.0.98.2:53787)
0619 08:56:59.281990 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:59.282031 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:59.282490 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:59.282726 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:59.282729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:59.283299 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:59.283484 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:59.283489 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:59.284176 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:59.284177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:59.284976 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:59.284978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:59.285361 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:59.285362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:59.285455 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:59.285880 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:59.285888 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:59.287314 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:59.288752 (+ 1438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:59.288760 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:59.288762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:59.288984 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:59.289167 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:59.289168 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:59.289168 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:59.289269 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:59.289482 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:59.289488 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:59.289630 (+ 142us) client_negotiation.cc:769] Sending connection context
0619 08:56:59.289719 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 08:56:59.289869 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
I20260619 08:56:59.290463 10972 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44529
I20260619 08:56:59.290540 10972 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:59.290700 10972 heartbeater.cc:511] Master 127.0.98.62:44529 requested a full tablet report, sending...
I20260619 08:56:59.291043 10647 ts_manager.cc:194] Registered new tserver with Master: f097d806482549f8a65419d503ad6de9 (127.0.98.2:33443)
I20260619 08:56:59.291548 10647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53787
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 08:56:59.308986 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44529
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:56:59.387985 10977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:56:59.388211 10977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:56:59.388233 10977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:56:59.389318 10977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:56:59.389351 10977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:56:59.389376 10977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 08:56:59.390738 10977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:56:59.390911 10977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:56:59.391083 10977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:56:59.395781 10982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:59.395915 10977 server_base.cc:1034] running on GCE node
W20260619 08:56:59.395763 10985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:56:59.395763 10983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:56:59.396219 10977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:56:59.396447 10977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:56:59.397585 10977 hybrid_clock.cc:648] HybridClock initialized: now 1781859419397570 us; error 29 us; skew 500 ppm
Jun 19 08:56:59 dist-test-slave-h4g5 krb5kdc[10602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:56:59.399909 10977 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 08:56:59.400352 10977 webserver.cc:460] Webserver started at http://127.0.98.3:35619/ using document root <none> and password file <none>
I20260619 08:56:59.400542 10977 fs_manager.cc:359] Metadata directory not provided
I20260619 08:56:59.400588 10977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:56:59.400702 10977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:56:59.401688 10977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-2/data/instance:
uuid: "c3568eb8fcd9494388f544c8936d6918"
format_stamp: "Formatted at 2026-06-19 08:56:59 on dist-test-slave-h4g5"
server_key: "845ea7818e9fb6d5b02c3c2bf07c5bb4"
server_key_iv: "99a2aeadd6af5e6abcca118559acacfd"
server_key_version: "encryptionkey@0"
I20260619 08:56:59.402009 10977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c3568eb8fcd9494388f544c8936d6918"
format_stamp: "Formatted at 2026-06-19 08:56:59 on dist-test-slave-h4g5"
server_key: "845ea7818e9fb6d5b02c3c2bf07c5bb4"
server_key_iv: "99a2aeadd6af5e6abcca118559acacfd"
server_key_version: "encryptionkey@0"
I20260619 08:56:59.403204 10977 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:56:59.403816 10992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:59.403992 10977 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:56:59.404070 10977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-2/wal
uuid: "c3568eb8fcd9494388f544c8936d6918"
format_stamp: "Formatted at 2026-06-19 08:56:59 on dist-test-slave-h4g5"
server_key: "845ea7818e9fb6d5b02c3c2bf07c5bb4"
server_key_iv: "99a2aeadd6af5e6abcca118559acacfd"
server_key_version: "encryptionkey@0"
I20260619 08:56:59.404155 10977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:56:59.419734 10977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:56:59.421586 10977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:56:59.421717 10977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:56:59.421943 10977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:56:59.422269 10977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:56:59.422312 10977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:59.422344 10977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:56:59.422369 10977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:56:59.427579 10977 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33701
I20260619 08:56:59.427625 11105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33701 every 8 connection(s)
I20260619 08:56:59.427918 10977 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
I20260619 08:56:59.433213 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 10977
I20260619 08:56:59.433315 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-2/wal/instance
I20260619 08:56:59.433476 392 external_mini_cluster.cc:1428] Setting key ae748daba4b59cff9a061601da56719e
Jun 19 08:56:59 dist-test-slave-h4g5 krb5kdc[10602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:56:59.436808 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:59.428589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55147 (local address 127.0.98.62:44529)
0619 08:56:59.428638 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:59.428640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:59.428828 (+ 188us) server_negotiation.cc:407] Connection header received
0619 08:56:59.429346 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:59.429347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:59.429359 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:59.429413 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:59.430335 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:59.430836 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:59.431871 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:59.432059 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:59.432144 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:59.434420 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:59.434421 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:59.434421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:59.434429 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:59.435649 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:59.436021 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:59.436022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:59.436022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:59.436071 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:59.436278 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:59.436279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:59.436279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:59.436402 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:59.436475 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:59.436669 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 08:56:59.436731 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 08:56:59.436957 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:59.428559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44529 (local address 127.0.98.3:55147)
0619 08:56:59.428715 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:59.428752 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:59.429226 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:59.429488 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:59.429492 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:59.430037 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:56:59.430229 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:59.430233 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:59.430923 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:59.430923 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:59.431788 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:59.431791 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:59.432151 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:59.432153 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:59.432278 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:59.432722 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:56:59.432731 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:56:59.434335 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:56:59.435731 (+ 1396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:59.435737 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:59.435739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:59.435957 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:59.436135 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:56:59.436136 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:56:59.436136 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:56:59.436230 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:56:59.436459 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:56:59.436462 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:56:59.436573 (+ 111us) client_negotiation.cc:769] Sending connection context
0619 08:56:59.436646 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 08:56:59.436765 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
I20260619 08:56:59.437384 11106 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44529
I20260619 08:56:59.437474 11106 heartbeater.cc:464] Registering TS with master...
I20260619 08:56:59.437618 11106 heartbeater.cc:511] Master 127.0.98.62:44529 requested a full tablet report, sending...
I20260619 08:56:59.437983 10647 ts_manager.cc:194] Registered new tserver with Master: c3568eb8fcd9494388f544c8936d6918 (127.0.98.3:33701)
I20260619 08:56:59.438318 10647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55147
I20260619 08:56:59.445132 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:56:59.451781 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:59.446318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52054 (local address 127.0.98.62:44529)
0619 08:56:59.446359 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:59.446361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:59.446472 (+ 111us) server_negotiation.cc:407] Connection header received
0619 08:56:59.446533 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:59.446534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:59.446544 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:59.446590 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:56:59.447102 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:59.447573 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:59.448338 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:59.448518 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:59.448571 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:59.449657 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:56:59.449659 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:56:59.449659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:56:59.449667 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:56:59.450705 (+ 1038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:59.451053 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:59.451055 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:59.451055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:59.451104 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:56:59.451271 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:56:59.451272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:56:59.451272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:56:59.451414 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:56:59.451498 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:59.451604 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 08:56:59.451682 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 08:56:59.453146 10647 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52054:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:56:59.454751 10647 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:56:59.460669 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:59.456860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33701 (local address 127.0.0.1:38456)
0619 08:56:59.457183 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:59.457192 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:59.457229 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:59.457430 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:59.457432 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:59.457434 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:59.457488 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:59.457489 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:59.458692 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:59.458693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:59.459669 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:59.459671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:59.460375 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:59.460376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:59.460451 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:59.460465 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 08:56:59.460530 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 08:56:59.460582 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":5,"thread_start_us":81,"threads_started":1}
I20260619 08:56:59.460760 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:59.457029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59472 (local address 127.0.98.2:33443)
0619 08:56:59.457522 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:59.457526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:59.457533 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:56:59.457560 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:59.457562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:59.457572 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:59.457639 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:59.457775 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:59.458553 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:59.459700 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:59.460452 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:59.460514 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:59.460567 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:59.460639 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 08:56:59.460676 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":72,"threads_started":1}
I20260619 08:56:59.461104 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:59.457077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38456 (local address 127.0.98.3:33701)
0619 08:56:59.457272 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:59.457277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:59.457288 (+ 11us) server_negotiation.cc:407] Connection header received
0619 08:56:59.457328 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:59.457330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:59.457345 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:59.457417 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:59.457567 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:59.458584 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:59.459751 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:59.460298 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:59.460367 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:59.460427 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:59.461010 (+ 583us) server_negotiation.cc:299] Negotiation successful
0619 08:56:59.461046 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":79,"threads_started":1}
I20260619 08:56:59.460755 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:59.456864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33443 (local address 127.0.0.1:59472)
0619 08:56:59.457237 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:59.457248 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:59.457303 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:59.457644 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:59.457645 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:59.457647 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:59.457690 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:59.457692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:59.458637 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:59.458638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:59.459595 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:59.459597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:59.460525 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:59.460526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:59.460578 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:59.460581 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:56:59.460629 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 08:56:59.460662 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":3,"thread_start_us":81,"threads_started":1}
I20260619 08:56:59.461503 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:59.456860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43685 (local address 127.0.0.1:40610)
0619 08:56:59.457001 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:59.457044 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:59.457112 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:59.457976 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:59.457977 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:59.457979 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:59.458022 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:59.458024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:59.459279 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:59.459280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:59.460325 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:59.460327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:59.461122 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:59.461123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:59.461353 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:59.461356 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:56:59.461401 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 08:56:59.461440 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":50,"threads_started":1}
I20260619 08:56:59.461890 11040 tablet_service.cc:1467] Processing CreateTablet for tablet 96b0823d99fa4a9f9a8729000efcc5e2 (DEFAULT_TABLE table=test-table [id=07910fbdb5364498849f25ca1693415d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:59.461992 10906 tablet_service.cc:1467] Processing CreateTablet for tablet 96b0823d99fa4a9f9a8729000efcc5e2 (DEFAULT_TABLE table=test-table [id=07910fbdb5364498849f25ca1693415d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:59.462251 11040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96b0823d99fa4a9f9a8729000efcc5e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:59.462332 10906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96b0823d99fa4a9f9a8729000efcc5e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:59.464768 11124 tablet_bootstrap.cc:492] T 96b0823d99fa4a9f9a8729000efcc5e2 P c3568eb8fcd9494388f544c8936d6918: Bootstrap starting.
I20260619 08:56:59.465055 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:59.457088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40610 (local address 127.0.98.1:43685)
0619 08:56:59.457769 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:59.457773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:59.457782 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:56:59.457819 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:59.457821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:59.457835 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:59.457902 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:59.458144 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:59.459177 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:59.460418 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:59.461044 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:59.461108 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:59.461158 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:59.461419 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 08:56:59.461461 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":75,"threads_started":1}
I20260619 08:56:59.465436 11124 tablet_bootstrap.cc:654] T 96b0823d99fa4a9f9a8729000efcc5e2 P c3568eb8fcd9494388f544c8936d6918: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:59.465687 10772 tablet_service.cc:1467] Processing CreateTablet for tablet 96b0823d99fa4a9f9a8729000efcc5e2 (DEFAULT_TABLE table=test-table [id=07910fbdb5364498849f25ca1693415d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:56:59.465878 11124 log.cc:824] T 96b0823d99fa4a9f9a8729000efcc5e2 P c3568eb8fcd9494388f544c8936d6918: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:59.465991 10772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96b0823d99fa4a9f9a8729000efcc5e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:56:59.466174 11123 tablet_bootstrap.cc:492] T 96b0823d99fa4a9f9a8729000efcc5e2 P f097d806482549f8a65419d503ad6de9: Bootstrap starting.
I20260619 08:56:59.466673 11123 tablet_bootstrap.cc:654] T 96b0823d99fa4a9f9a8729000efcc5e2 P f097d806482549f8a65419d503ad6de9: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:59.467085 11123 log.cc:824] T 96b0823d99fa4a9f9a8729000efcc5e2 P f097d806482549f8a65419d503ad6de9: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:59.467620 11124 tablet_bootstrap.cc:492] T 96b0823d99fa4a9f9a8729000efcc5e2 P c3568eb8fcd9494388f544c8936d6918: No bootstrap required, opened a new log
I20260619 08:56:59.467681 11124 ts_tablet_manager.cc:1397] T 96b0823d99fa4a9f9a8729000efcc5e2 P c3568eb8fcd9494388f544c8936d6918: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 08:56:59.467792 11123 tablet_bootstrap.cc:492] T 96b0823d99fa4a9f9a8729000efcc5e2 P f097d806482549f8a65419d503ad6de9: No bootstrap required, opened a new log
I20260619 08:56:59.467864 11123 ts_tablet_manager.cc:1397] T 96b0823d99fa4a9f9a8729000efcc5e2 P f097d806482549f8a65419d503ad6de9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:56:59.468853 11127 tablet_bootstrap.cc:492] T 96b0823d99fa4a9f9a8729000efcc5e2 P cc9c5351059541e7b28d896485e91b6e: Bootstrap starting.
I20260619 08:56:59.469514 11127 tablet_bootstrap.cc:654] T 96b0823d99fa4a9f9a8729000efcc5e2 P cc9c5351059541e7b28d896485e91b6e: Neither blocks nor log segments found. Creating new log.
I20260619 08:56:59.469539 11124 raft_consensus.cc:348] T 96b0823d99fa4a9f9a8729000efcc5e2 P c3568eb8fcd9494388f544c8936d6918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3568eb8fcd9494388f544c8936d6918" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33701 } } peers { permanent_uuid: "f097d806482549f8a65419d503ad6de9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33443 } } peers { permanent_uuid: "cc9c5351059541e7b28d896485e91b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43685 } }
I20260619 08:56:59.469722 11124 raft_consensus.cc:374] T 96b0823d99fa4a9f9a8729000efcc5e2 P c3568eb8fcd9494388f544c8936d6918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:59.469647 11123 raft_consensus.cc:348] T 96b0823d99fa4a9f9a8729000efcc5e2 P f097d806482549f8a65419d503ad6de9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3568eb8fcd9494388f544c8936d6918" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33701 } } peers { permanent_uuid: "f097d806482549f8a65419d503ad6de9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33443 } } peers { permanent_uuid: "cc9c5351059541e7b28d896485e91b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43685 } }
I20260619 08:56:59.469766 11124 raft_consensus.cc:729] T 96b0823d99fa4a9f9a8729000efcc5e2 P c3568eb8fcd9494388f544c8936d6918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3568eb8fcd9494388f544c8936d6918, State: Initialized, Role: FOLLOWER
I20260619 08:56:59.469800 11123 raft_consensus.cc:374] T 96b0823d99fa4a9f9a8729000efcc5e2 P f097d806482549f8a65419d503ad6de9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:59.469832 11123 raft_consensus.cc:729] T 96b0823d99fa4a9f9a8729000efcc5e2 P f097d806482549f8a65419d503ad6de9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f097d806482549f8a65419d503ad6de9, State: Initialized, Role: FOLLOWER
I20260619 08:56:59.469904 11124 consensus_queue.cc:260] T 96b0823d99fa4a9f9a8729000efcc5e2 P c3568eb8fcd9494388f544c8936d6918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3568eb8fcd9494388f544c8936d6918" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33701 } } peers { permanent_uuid: "f097d806482549f8a65419d503ad6de9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33443 } } peers { permanent_uuid: "cc9c5351059541e7b28d896485e91b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43685 } }
I20260619 08:56:59.469944 11123 consensus_queue.cc:260] T 96b0823d99fa4a9f9a8729000efcc5e2 P f097d806482549f8a65419d503ad6de9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3568eb8fcd9494388f544c8936d6918" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33701 } } peers { permanent_uuid: "f097d806482549f8a65419d503ad6de9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33443 } } peers { permanent_uuid: "cc9c5351059541e7b28d896485e91b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43685 } }
I20260619 08:56:59.470079 11127 log.cc:824] T 96b0823d99fa4a9f9a8729000efcc5e2 P cc9c5351059541e7b28d896485e91b6e: Log is configured to *not* fsync() on all Append() calls
I20260619 08:56:59.470207 11124 ts_tablet_manager.cc:1428] T 96b0823d99fa4a9f9a8729000efcc5e2 P c3568eb8fcd9494388f544c8936d6918: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 08:56:59.470208 11123 ts_tablet_manager.cc:1428] T 96b0823d99fa4a9f9a8729000efcc5e2 P f097d806482549f8a65419d503ad6de9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:56:59.470249 10972 heartbeater.cc:503] Master 127.0.98.62:44529 was elected leader, sending a full tablet report...
I20260619 08:56:59.470343 11106 heartbeater.cc:503] Master 127.0.98.62:44529 was elected leader, sending a full tablet report...
I20260619 08:56:59.470856 11127 tablet_bootstrap.cc:492] T 96b0823d99fa4a9f9a8729000efcc5e2 P cc9c5351059541e7b28d896485e91b6e: No bootstrap required, opened a new log
I20260619 08:56:59.470924 11127 ts_tablet_manager.cc:1397] T 96b0823d99fa4a9f9a8729000efcc5e2 P cc9c5351059541e7b28d896485e91b6e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:56:59.472589 11127 raft_consensus.cc:348] T 96b0823d99fa4a9f9a8729000efcc5e2 P cc9c5351059541e7b28d896485e91b6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3568eb8fcd9494388f544c8936d6918" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33701 } } peers { permanent_uuid: "f097d806482549f8a65419d503ad6de9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33443 } } peers { permanent_uuid: "cc9c5351059541e7b28d896485e91b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43685 } }
I20260619 08:56:59.472775 11127 raft_consensus.cc:374] T 96b0823d99fa4a9f9a8729000efcc5e2 P cc9c5351059541e7b28d896485e91b6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:56:59.472819 11127 raft_consensus.cc:729] T 96b0823d99fa4a9f9a8729000efcc5e2 P cc9c5351059541e7b28d896485e91b6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc9c5351059541e7b28d896485e91b6e, State: Initialized, Role: FOLLOWER
I20260619 08:56:59.472934 11127 consensus_queue.cc:260] T 96b0823d99fa4a9f9a8729000efcc5e2 P cc9c5351059541e7b28d896485e91b6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3568eb8fcd9494388f544c8936d6918" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33701 } } peers { permanent_uuid: "f097d806482549f8a65419d503ad6de9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33443 } } peers { permanent_uuid: "cc9c5351059541e7b28d896485e91b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43685 } }
I20260619 08:56:59.473169 11127 ts_tablet_manager.cc:1428] T 96b0823d99fa4a9f9a8729000efcc5e2 P cc9c5351059541e7b28d896485e91b6e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 08:56:59.473201 10838 heartbeater.cc:503] Master 127.0.98.62:44529 was elected leader, sending a full tablet report...
I20260619 08:56:59.525857 11128 raft_consensus.cc:482] T 96b0823d99fa4a9f9a8729000efcc5e2 P c3568eb8fcd9494388f544c8936d6918 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:56:59.525986 11128 raft_consensus.cc:504] T 96b0823d99fa4a9f9a8729000efcc5e2 P c3568eb8fcd9494388f544c8936d6918 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3568eb8fcd9494388f544c8936d6918" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33701 } } peers { permanent_uuid: "f097d806482549f8a65419d503ad6de9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33443 } } peers { permanent_uuid: "cc9c5351059541e7b28d896485e91b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43685 } }
I20260619 08:56:59.526384 11128 leader_election.cc:288] T 96b0823d99fa4a9f9a8729000efcc5e2 P c3568eb8fcd9494388f544c8936d6918 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f097d806482549f8a65419d503ad6de9 (127.0.98.2:33443), cc9c5351059541e7b28d896485e91b6e (127.0.98.1:43685)
I20260619 08:56:59.530063 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:59.526615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57923 (local address 127.0.98.2:33443)
0619 08:56:59.526664 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:59.526666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:59.526676 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:56:59.526707 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:59.526708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:59.526718 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:59.526773 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:59.526937 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:59.527679 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:59.529130 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:59.529704 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:59.529772 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:59.529782 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:59.529946 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 08:56:59.529987 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 08:56:59.530063 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:59.526632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50515 (local address 127.0.98.1:43685)
0619 08:56:59.526674 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:59.526675 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:59.526839 (+ 164us) server_negotiation.cc:407] Connection header received
0619 08:56:59.526865 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:59.526866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:59.526876 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:59.526922 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:56:59.527048 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:59.527754 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:59.529129 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:59.529704 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:59.529775 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:59.529782 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:59.529946 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 08:56:59.529987 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 08:56:59.530094 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:59.526545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33443 (local address 127.0.98.3:57923)
0619 08:56:59.526604 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:59.526615 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:59.526658 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:59.526797 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:59.526798 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:59.526799 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:59.526872 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:59.526874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:59.527775 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:59.527776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:59.529037 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:59.529041 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:59.529788 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:59.529789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:59.529878 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:59.529880 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:56:59.529932 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 08:56:59.529981 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":14}
I20260619 08:56:59.530102 11132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:59.526551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43685 (local address 127.0.98.3:50515)
0619 08:56:59.526742 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:59.526764 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 08:56:59.526819 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:56:59.526935 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:56:59.526936 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:56:59.526937 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:56:59.526993 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:59.526994 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:59.527828 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:59.527829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:59.529037 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:56:59.529041 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:56:59.529789 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:56:59.529790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:56:59.529875 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:59.529879 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:56:59.529932 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 08:56:59.529981 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":18,"thread_start_us":124,"threads_started":1}
I20260619 08:56:59.530472 10792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96b0823d99fa4a9f9a8729000efcc5e2" candidate_uuid: "c3568eb8fcd9494388f544c8936d6918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc9c5351059541e7b28d896485e91b6e" is_pre_election: true
I20260619 08:56:59.530539 10926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96b0823d99fa4a9f9a8729000efcc5e2" candidate_uuid: "c3568eb8fcd9494388f544c8936d6918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f097d806482549f8a65419d503ad6de9" is_pre_election: true
I20260619 08:56:59.530647 10792 raft_consensus.cc:2454] T 96b0823d99fa4a9f9a8729000efcc5e2 P cc9c5351059541e7b28d896485e91b6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3568eb8fcd9494388f544c8936d6918 in term 0.
I20260619 08:56:59.530650 10926 raft_consensus.cc:2454] T 96b0823d99fa4a9f9a8729000efcc5e2 P f097d806482549f8a65419d503ad6de9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3568eb8fcd9494388f544c8936d6918 in term 0.
I20260619 08:56:59.530889 10995 leader_election.cc:302] T 96b0823d99fa4a9f9a8729000efcc5e2 P c3568eb8fcd9494388f544c8936d6918 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3568eb8fcd9494388f544c8936d6918, cc9c5351059541e7b28d896485e91b6e; no voters:
I20260619 08:56:59.531014 11128 raft_consensus.cc:2781] T 96b0823d99fa4a9f9a8729000efcc5e2 P c3568eb8fcd9494388f544c8936d6918 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:56:59.531070 11128 raft_consensus.cc:482] T 96b0823d99fa4a9f9a8729000efcc5e2 P c3568eb8fcd9494388f544c8936d6918 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:56:59.531092 11128 raft_consensus.cc:3037] T 96b0823d99fa4a9f9a8729000efcc5e2 P c3568eb8fcd9494388f544c8936d6918 [term 0 FOLLOWER]: Advancing to term 1
W20260619 08:56:59.531751 10973 tablet.cc:2369] T 96b0823d99fa4a9f9a8729000efcc5e2 P f097d806482549f8a65419d503ad6de9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:56:59.531845 11128 raft_consensus.cc:504] T 96b0823d99fa4a9f9a8729000efcc5e2 P c3568eb8fcd9494388f544c8936d6918 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3568eb8fcd9494388f544c8936d6918" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33701 } } peers { permanent_uuid: "f097d806482549f8a65419d503ad6de9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33443 } } peers { permanent_uuid: "cc9c5351059541e7b28d896485e91b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43685 } }
I20260619 08:56:59.531982 11128 leader_election.cc:288] T 96b0823d99fa4a9f9a8729000efcc5e2 P c3568eb8fcd9494388f544c8936d6918 [CANDIDATE]: Term 1 election: Requested vote from peers f097d806482549f8a65419d503ad6de9 (127.0.98.2:33443), cc9c5351059541e7b28d896485e91b6e (127.0.98.1:43685)
I20260619 08:56:59.532137 10926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96b0823d99fa4a9f9a8729000efcc5e2" candidate_uuid: "c3568eb8fcd9494388f544c8936d6918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f097d806482549f8a65419d503ad6de9"
I20260619 08:56:59.532159 10792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96b0823d99fa4a9f9a8729000efcc5e2" candidate_uuid: "c3568eb8fcd9494388f544c8936d6918" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc9c5351059541e7b28d896485e91b6e"
I20260619 08:56:59.532215 10926 raft_consensus.cc:3037] T 96b0823d99fa4a9f9a8729000efcc5e2 P f097d806482549f8a65419d503ad6de9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:59.532231 10792 raft_consensus.cc:3037] T 96b0823d99fa4a9f9a8729000efcc5e2 P cc9c5351059541e7b28d896485e91b6e [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:56:59.532882 10926 raft_consensus.cc:2454] T 96b0823d99fa4a9f9a8729000efcc5e2 P f097d806482549f8a65419d503ad6de9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3568eb8fcd9494388f544c8936d6918 in term 1.
I20260619 08:56:59.532922 10792 raft_consensus.cc:2454] T 96b0823d99fa4a9f9a8729000efcc5e2 P cc9c5351059541e7b28d896485e91b6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3568eb8fcd9494388f544c8936d6918 in term 1.
I20260619 08:56:59.533044 10994 leader_election.cc:302] T 96b0823d99fa4a9f9a8729000efcc5e2 P c3568eb8fcd9494388f544c8936d6918 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3568eb8fcd9494388f544c8936d6918, f097d806482549f8a65419d503ad6de9; no voters:
I20260619 08:56:59.533119 11128 raft_consensus.cc:2781] T 96b0823d99fa4a9f9a8729000efcc5e2 P c3568eb8fcd9494388f544c8936d6918 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:56:59.533267 11128 raft_consensus.cc:686] T 96b0823d99fa4a9f9a8729000efcc5e2 P c3568eb8fcd9494388f544c8936d6918 [term 1 LEADER]: Becoming Leader. State: Replica: c3568eb8fcd9494388f544c8936d6918, State: Running, Role: LEADER
I20260619 08:56:59.533353 11128 consensus_queue.cc:237] T 96b0823d99fa4a9f9a8729000efcc5e2 P c3568eb8fcd9494388f544c8936d6918 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3568eb8fcd9494388f544c8936d6918" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33701 } } peers { permanent_uuid: "f097d806482549f8a65419d503ad6de9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33443 } } peers { permanent_uuid: "cc9c5351059541e7b28d896485e91b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43685 } }
I20260619 08:56:59.534130 10646 catalog_manager.cc:5495] T 96b0823d99fa4a9f9a8729000efcc5e2 P c3568eb8fcd9494388f544c8936d6918 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3568eb8fcd9494388f544c8936d6918 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c3568eb8fcd9494388f544c8936d6918" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3568eb8fcd9494388f544c8936d6918" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f097d806482549f8a65419d503ad6de9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc9c5351059541e7b28d896485e91b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43685 } health_report { overall_health: UNKNOWN } } }
I20260619 08:56:59.552628 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:59.550313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38460 (local address 127.0.98.3:33701)
0619 08:56:59.550388 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 08:56:59.550390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:56:59.550412 (+ 22us) server_negotiation.cc:407] Connection header received
0619 08:56:59.550444 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:56:59.550445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:56:59.550464 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:56:59.550518 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:56:59.550715 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:59.551198 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:59.551957 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:56:59.552167 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:56:59.552216 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:56:59.552327 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:56:59.552427 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:56:59.552484 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 08:56:59.552513 (+ 29us) server_negotiation.cc:299] Negotiation successful
0619 08:56:59.552531 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 08:56:59.554656 10926 raft_consensus.cc:1261] T 96b0823d99fa4a9f9a8729000efcc5e2 P f097d806482549f8a65419d503ad6de9 [term 1 FOLLOWER]: Refusing update from remote peer c3568eb8fcd9494388f544c8936d6918: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:56:59.554996 10792 raft_consensus.cc:1261] T 96b0823d99fa4a9f9a8729000efcc5e2 P cc9c5351059541e7b28d896485e91b6e [term 1 FOLLOWER]: Refusing update from remote peer c3568eb8fcd9494388f544c8936d6918: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:56:59.555011 11128 consensus_queue.cc:1035] T 96b0823d99fa4a9f9a8729000efcc5e2 P c3568eb8fcd9494388f544c8936d6918 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f097d806482549f8a65419d503ad6de9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:56:59.555292 11128 consensus_queue.cc:1035] T 96b0823d99fa4a9f9a8729000efcc5e2 P c3568eb8fcd9494388f544c8936d6918 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc9c5351059541e7b28d896485e91b6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:56:59.557897 11137 mvcc.cc:205] Tried to move back new op lower bound from 7298496182492598272 to 7298496182409277440. Current Snapshot: MvccSnapshot[applied={T|T < 7298496182492598272}]
I20260619 08:56:59.558828 11138 mvcc.cc:205] Tried to move back new op lower bound from 7298496182492598272 to 7298496182409277440. Current Snapshot: MvccSnapshot[applied={T|T < 7298496182492598272 or (T in {7298496182492598272})}]
I20260619 08:56:59.560058 10646 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:56:59.560181 10646 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:56:59.560863 10646 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 900887889557490fb9ab5308a07b83fc: Sending DeleteTablet for 3 replicas of tablet 96b0823d99fa4a9f9a8729000efcc5e2
I20260619 08:56:59.561151 11040 tablet_service.cc:1514] Processing DeleteTablet for tablet 96b0823d99fa4a9f9a8729000efcc5e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38456
I20260619 08:56:59.561179 10906 tablet_service.cc:1514] Processing DeleteTablet for tablet 96b0823d99fa4a9f9a8729000efcc5e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59472
I20260619 08:56:59.561219 10772 tablet_service.cc:1514] Processing DeleteTablet for tablet 96b0823d99fa4a9f9a8729000efcc5e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:56:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40610
I20260619 08:56:59.561410 11147 tablet_replica.cc:330] T 96b0823d99fa4a9f9a8729000efcc5e2 P c3568eb8fcd9494388f544c8936d6918: stopping tablet replica
I20260619 08:56:59.561514 11147 raft_consensus.cc:2229] T 96b0823d99fa4a9f9a8729000efcc5e2 P c3568eb8fcd9494388f544c8936d6918 [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:56:59.561604 11147 raft_consensus.cc:2258] T 96b0823d99fa4a9f9a8729000efcc5e2 P c3568eb8fcd9494388f544c8936d6918 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:59.561719 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 10709
I20260619 08:56:59.561751 11146 tablet_replica.cc:330] T 96b0823d99fa4a9f9a8729000efcc5e2 P f097d806482549f8a65419d503ad6de9: stopping tablet replica
I20260619 08:56:59.561841 11146 raft_consensus.cc:2229] T 96b0823d99fa4a9f9a8729000efcc5e2 P f097d806482549f8a65419d503ad6de9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:56:59.561929 11146 raft_consensus.cc:2258] T 96b0823d99fa4a9f9a8729000efcc5e2 P f097d806482549f8a65419d503ad6de9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:56:59.562321 11147 ts_tablet_manager.cc:1905] T 96b0823d99fa4a9f9a8729000efcc5e2 P c3568eb8fcd9494388f544c8936d6918: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:59.563767 11147 ts_tablet_manager.cc:1918] T 96b0823d99fa4a9f9a8729000efcc5e2 P c3568eb8fcd9494388f544c8936d6918: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:56:59.563865 11146 ts_tablet_manager.cc:1905] T 96b0823d99fa4a9f9a8729000efcc5e2 P f097d806482549f8a65419d503ad6de9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:56:59.563892 11147 log.cc:1196] T 96b0823d99fa4a9f9a8729000efcc5e2 P c3568eb8fcd9494388f544c8936d6918: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-2/wal/wals/96b0823d99fa4a9f9a8729000efcc5e2
I20260619 08:56:59.564266 11147 ts_tablet_manager.cc:1939] T 96b0823d99fa4a9f9a8729000efcc5e2 P c3568eb8fcd9494388f544c8936d6918: Deleting consensus metadata
I20260619 08:56:59.564702 10634 catalog_manager.cc:4841] TS c3568eb8fcd9494388f544c8936d6918 (127.0.98.3:33701): tablet 96b0823d99fa4a9f9a8729000efcc5e2 (table test-table [id=07910fbdb5364498849f25ca1693415d]) successfully deleted
I20260619 08:56:59.565412 11146 ts_tablet_manager.cc:1918] T 96b0823d99fa4a9f9a8729000efcc5e2 P f097d806482549f8a65419d503ad6de9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:56:59.565475 11146 log.cc:1196] T 96b0823d99fa4a9f9a8729000efcc5e2 P f097d806482549f8a65419d503ad6de9: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859361540516-392-0/minicluster-data/ts-1/wal/wals/96b0823d99fa4a9f9a8729000efcc5e2
I20260619 08:56:59.565707 11146 ts_tablet_manager.cc:1939] T 96b0823d99fa4a9f9a8729000efcc5e2 P f097d806482549f8a65419d503ad6de9: Deleting consensus metadata
I20260619 08:56:59.566059 10632 catalog_manager.cc:4841] TS f097d806482549f8a65419d503ad6de9 (127.0.98.2:33443): tablet 96b0823d99fa4a9f9a8729000efcc5e2 (table test-table [id=07910fbdb5364498849f25ca1693415d]) successfully deleted
W20260619 08:56:59.569272 10633 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43685 (error 108)
I20260619 08:56:59.569757 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:56:59.569643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43685 (local address 127.0.0.1:40620)
0619 08:56:59.569693 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 08:56:59.569726 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 08:56:59.569844 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 10843
W20260619 08:56:59.569883 10633 catalog_manager.cc:4584] TS cc9c5351059541e7b28d896485e91b6e (127.0.98.1:43685): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 96b0823d99fa4a9f9a8729000efcc5e2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43685: connect: Connection refused (error 111)
I20260619 08:56:59.574477 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 10977
I20260619 08:56:59.578624 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 10615
2026-06-19T08:56:59Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1828 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:56:59 dist-test-slave-h4g5 krb5kdc[11152](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:56:59 dist-test-slave-h4g5 krb5kdc[11152](info): set up 2 sockets
Jun 19 08:56:59 dist-test-slave-h4g5 krb5kdc[11152](info): commencing operation
krb5kdc: starting...
W20260619 08:57:00.540778 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 08:57:00 dist-test-slave-h4g5 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:57:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:57:00Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260619 08:57:00.701372 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task5ocqKJ/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/postgres -l logfile start
2026-06-19 08:57:03.083 UTC [11182] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 08:57:03.083 UTC [11182] LOG: listening on IPv4 address "127.0.98.20", port 59245
2026-06-19 08:57:03.086 UTC [11182] LOG: listening on Unix socket "/tmp/.s.PGSQL.59245"
2026-06-19 08:57:03.106 UTC [11185] LOG: database system was shut down at 2026-06-19 08:57:01 UTC
2026-06-19 08:57:03.110 UTC [11182] LOG: database system is ready to accept connections
I20260619 08:57:03.988623 392 mini_postgres.cc:96] Postgres bound to 59245
2026-06-19 08:57:03.994 UTC [11193] FATAL: database "slave" does not exist
127.0.98.20:59245 - accepting connections
I20260619 08:57:03.994830 392 mini_ranger.cc:161] Starting Ranger...
I20260619 08:57:04.008855 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 08:57:04.476540 392 mini_ranger.cc:87] Created ranger Postgres database
I20260619 08:57:04.476608 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ranger-admin
2026-06-19 08:57:04,737 [I] DB FLAVOR :POSTGRES
2026-06-19 08:57:04,738 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 08:57:04,738 [I] Checking connection..
2026-06-19 08:57:04,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59245/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 08:57:04,984 [I] Checking connection passed.
2026-06-19 08:57:04,984 [I] --------- Verifying version history table ---------
2026-06-19 08:57:04,984 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59245/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 08:57:05,242 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 08:57:05,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59245/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 08:57:05,484 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 08:57:05,484 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 08:57:05,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59245/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 08:57:05.713 UTC [11257] WARNING: there is no transaction in progress
2026-06-19 08:57:05,718 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 08:57:05,719 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59245/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 08:57:05,980 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 08:57:05,980 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 08:57:05,980 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59245/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 08:57:06,205 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59245/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 08:57:06,442 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59245/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-h4g5.c.gcp-upstream.internal','N') ;"
2026-06-19 08:57:06,693 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59245/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 08:57:06,924 [I] Table x_portal_user does not exist in database ranger
2026-06-19 08:57:06,925 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59245/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 08:57:07,162 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 08:57:07,162 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59245/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 08:57:07,396 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59245/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 08:57:07,646 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 08:57:07,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59245/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 08:57:08.603 UTC [11365] WARNING: there is no transaction in progress
2026-06-19 08:57:08.619 UTC [11365] WARNING: there is no transaction in progress
2026-06-19 08:57:08.635 UTC [11365] WARNING: there is no transaction in progress
2026-06-19 08:57:08.650 UTC [11365] WARNING: there is no transaction in progress
2026-06-19 08:57:08.665 UTC [11365] WARNING: there is no transaction in progress
2026-06-19 08:57:08.684 UTC [11365] WARNING: there is no transaction in progress
2026-06-19 08:57:08.747 UTC [11365] WARNING: there is no transaction in progress
2026-06-19 08:57:08.754 UTC [11365] WARNING: there is no transaction in progress
2026-06-19 08:57:08.769 UTC [11365] WARNING: there is no transaction in progress
2026-06-19 08:57:08.779 UTC [11365] WARNING: there is no transaction in progress
2026-06-19 08:57:08.794 UTC [11365] WARNING: there is no transaction in progress
2026-06-19 08:57:08.807 UTC [11365] WARNING: there is no transaction in progress
2026-06-19 08:57:08.817 UTC [11365] WARNING: there is no transaction in progress
2026-06-19 08:57:08.823 UTC [11365] WARNING: there is no transaction in progress
2026-06-19 08:57:08.830 UTC [11365] WARNING: there is no transaction in progress
2026-06-19 08:57:08.836 UTC [11365] WARNING: there is no transaction in progress
2026-06-19 08:57:09,430 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 08:57:09,430 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59245/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 08:57:09,656 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 08:57:09,656 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59245/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 08:57:09,909 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 08:57:09,909 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59245/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 08:57:10,132 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 08:57:10,132 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59245/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 08:57:10,373 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59245/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 08:57:10,623 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59245/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-h4g5.c.gcp-upstream.internal';"
2026-06-19 08:57:10,866 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 08:57:10,866 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59245/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 08:57:11,115 [I] DB_PATCHES have already been applied
I20260619 08:57:11.120311 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task5ocqKJ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 08:57:11.120381 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260619 08:57:11.123164 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:37207
Jun 19, 2026 8:57:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 8:57:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 8:57:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 8:57:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 8:57:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 8:57:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 8:57:12 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 08:57:12.118311 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.123s user 0.002s sys 0.007s
Jun 19, 2026 8:57:31 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 8:57:31 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 19, 2026 8:57:31 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 8:57:31 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 8:57:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 8:57:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 8:57:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 8:57:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 8:57:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 8:57:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 8:57:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 8:57:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 8:57:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 8:57:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 8:57:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 8:57:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 8:57:31 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 8:57:32 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 08:57:32.727396 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 08:57:32 dist-test-slave-h4g5 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859452, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 08:57:32.799203 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 08:57:32.807456 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 08:57:33.603134 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 08:57:33.603214 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ranger-kms
I20260619 08:57:33.603256 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:59245
2026-06-19 08:57:33,763 [I] DB FLAVOR :POSTGRES
2026-06-19 08:57:33,763 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 08:57:33,763 [I] Checking connection
2026-06-19 08:57:33,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59245/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 08:57:34,009 [I] connection success
2026-06-19 08:57:34,009 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 08:57:34,009 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 08:57:34,010 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59245/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 08:57:34,250 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 08:57:34,251 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 08:57:34,251 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 08:57:34,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59245/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 08:57:34,514 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 08:57:34.995373 392 mini_ranger_kms.cc:310] Created kms service
I20260619 08:57:35.179592 392 mini_ranger_kms.cc:326] Created kudu user
I20260619 08:57:35.222741 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 08:57:35 dist-test-slave-h4g5 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859452, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 08:57:35.598302 392 mini_ranger_kms.cc:384] Added ranger policy
I20260619 08:57:35.598416 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 08:57:35.598438 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260619 08:57:35.601517 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11545
I20260619 08:57:35.601640 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:60323
Jun 19, 2026 8:57:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 8:57:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 8:57:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 8:57:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 8:57:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 8:57:36 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 08:57:37.615767 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.817s user 0.000s sys 0.142s
I20260619 08:57:37.615862 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 08:57:37.615937 392 mini_ranger_kms.cc:401] 127.0.98.20:60323/kms/v1/keys
Jun 19 08:57:37 dist-test-slave-h4g5 krb5kdc[11152](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781859457, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 08:57:38 dist-test-slave-h4g5 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859457, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 08:57:38 dist-test-slave-h4g5 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859457, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 08:57:40 dist-test-slave-h4g5 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 08:57:40.848408 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43187
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43187
--ranger_config_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60323/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:57:40.929548 11596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:57:40.929754 11596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:57:40.929773 11596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:57:40.930888 11596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:57:40.930929 11596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:57:40.930943 11596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:57:40.930953 11596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:57:40.930963 11596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:57:40.932538 11596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:60323/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43187
--ranger_config_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:57:40.932757 11596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:57:40.932966 11596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:57:40.938962 11604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:57:40.939126 11601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:57:40.939002 11602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:57:40.939033 11596 server_base.cc:1034] running on GCE node
I20260619 08:57:40.939535 11596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:57:40.939867 11596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:57:40.941033 11596 hybrid_clock.cc:648] HybridClock initialized: now 1781859460941013 us; error 31 us; skew 500 ppm
Jun 19 08:57:40 dist-test-slave-h4g5 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:57:40.943854 11596 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 08:57:40.944306 11596 webserver.cc:460] Webserver started at http://127.0.98.62:34161/ using document root <none> and password file <none>
I20260619 08:57:40.944507 11596 fs_manager.cc:359] Metadata directory not provided
I20260619 08:57:40.944546 11596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:57:40.944677 11596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 08:57:40 dist-test-slave-h4g5 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 08:57:40.992305 11596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "0c83e311c18d491eb35a48825a069e71"
format_stamp: "Formatted at 2026-06-19 08:57:40 on dist-test-slave-h4g5"
server_key: "b966b3a72062119b6fdbd22bda68f62e"
server_key_iv: "9458e87e79601b98a21985bc806022bf"
server_key_version: "kuduclusterkey@0"
I20260619 08:57:40.992674 11596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "0c83e311c18d491eb35a48825a069e71"
format_stamp: "Formatted at 2026-06-19 08:57:40 on dist-test-slave-h4g5"
server_key: "b966b3a72062119b6fdbd22bda68f62e"
server_key_iv: "9458e87e79601b98a21985bc806022bf"
server_key_version: "kuduclusterkey@0"
I20260619 08:57:40.994078 11596 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:57:41.006173 11611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:57:41.006367 11596 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:57:41.006457 11596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "0c83e311c18d491eb35a48825a069e71"
format_stamp: "Formatted at 2026-06-19 08:57:40 on dist-test-slave-h4g5"
server_key: "b966b3a72062119b6fdbd22bda68f62e"
server_key_iv: "9458e87e79601b98a21985bc806022bf"
server_key_version: "kuduclusterkey@0"
I20260619 08:57:41.006558 11596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:57:41.017304 11596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:57:41.019201 11596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:57:41.019346 11596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:57:41.024503 11596 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43187
I20260619 08:57:41.024537 11663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43187 every 8 connection(s)
I20260619 08:57:41.024896 11596 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:57:41.025540 11664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:57:41.027858 11664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c83e311c18d491eb35a48825a069e71: Bootstrap starting.
I20260619 08:57:41.028503 11664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c83e311c18d491eb35a48825a069e71: Neither blocks nor log segments found. Creating new log.
I20260619 08:57:41.028761 11664 log.cc:824] T 00000000000000000000000000000000 P 0c83e311c18d491eb35a48825a069e71: Log is configured to *not* fsync() on all Append() calls
I20260619 08:57:41.029421 11664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c83e311c18d491eb35a48825a069e71: No bootstrap required, opened a new log
I20260619 08:57:41.031457 11664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c83e311c18d491eb35a48825a069e71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c83e311c18d491eb35a48825a069e71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43187 } }
I20260619 08:57:41.031612 11664 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c83e311c18d491eb35a48825a069e71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:57:41.031633 11664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c83e311c18d491eb35a48825a069e71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c83e311c18d491eb35a48825a069e71, State: Initialized, Role: FOLLOWER
I20260619 08:57:41.031766 11664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c83e311c18d491eb35a48825a069e71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c83e311c18d491eb35a48825a069e71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43187 } }
I20260619 08:57:41.031836 11664 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c83e311c18d491eb35a48825a069e71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:57:41.031867 11664 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c83e311c18d491eb35a48825a069e71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:57:41.031908 11664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c83e311c18d491eb35a48825a069e71 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:57:41.032610 11664 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c83e311c18d491eb35a48825a069e71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c83e311c18d491eb35a48825a069e71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43187 } }
I20260619 08:57:41.032719 11664 leader_election.cc:302] T 00000000000000000000000000000000 P 0c83e311c18d491eb35a48825a069e71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c83e311c18d491eb35a48825a069e71; no voters:
I20260619 08:57:41.032877 11664 leader_election.cc:288] T 00000000000000000000000000000000 P 0c83e311c18d491eb35a48825a069e71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:57:41.032929 11667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c83e311c18d491eb35a48825a069e71 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:57:41.033099 11667 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c83e311c18d491eb35a48825a069e71 [term 1 LEADER]: Becoming Leader. State: Replica: 0c83e311c18d491eb35a48825a069e71, State: Running, Role: LEADER
I20260619 08:57:41.033177 11664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c83e311c18d491eb35a48825a069e71 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:57:41.033252 11667 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c83e311c18d491eb35a48825a069e71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c83e311c18d491eb35a48825a069e71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43187 } }
I20260619 08:57:41.033761 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 11596
I20260619 08:57:41.033866 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 08:57:41.034060 392 external_mini_cluster.cc:1428] Setting key 934c998d0a483bb145f1f801f042dc04
I20260619 08:57:41.034722 11664 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 08:57:41.034953 11668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c83e311c18d491eb35a48825a069e71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c83e311c18d491eb35a48825a069e71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c83e311c18d491eb35a48825a069e71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43187 } } }
I20260619 08:57:41.035033 11668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c83e311c18d491eb35a48825a069e71 [sys.catalog]: This master's current role is: LEADER
I20260619 08:57:41.037825 11685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:57:41.038563 11669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c83e311c18d491eb35a48825a069e71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c83e311c18d491eb35a48825a069e71. Latest consensus state: current_term: 1 leader_uuid: "0c83e311c18d491eb35a48825a069e71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c83e311c18d491eb35a48825a069e71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43187 } } }
I20260619 08:57:41.038699 11669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c83e311c18d491eb35a48825a069e71 [sys.catalog]: This master's current role is: LEADER
I20260619 08:57:41.038913 11685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:57:41.042606 11685 catalog_manager.cc:1347] Generated new cluster ID: e8079f3bec314bd2aa7e2acf0f118d4b
I20260619 08:57:41.042666 11685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 08:57:41 dist-test-slave-h4g5 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:57:41.053095 11683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:57:41.034534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43354 (local address 127.0.98.62:43187)
0619 08:57:41.034691 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 08:57:41.034701 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 08:57:41.037547 (+ 2846us) server_negotiation.cc:407] Connection header received
0619 08:57:41.037957 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:57:41.037966 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:57:41.038081 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:57:41.038295 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:57:41.039062 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:57:41.041099 (+ 2037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:57:41.042191 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:57:41.042848 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:57:41.042959 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:57:41.049321 (+ 6362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:57:41.049328 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:57:41.049328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:57:41.049342 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:57:41.051368 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:57:41.051768 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:57:41.051771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:57:41.051772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:57:41.051877 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:57:41.052064 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:57:41.052065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:57:41.052066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:57:41.052457 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:57:41.052621 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 08:57:41.052723 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 08:57:41.052858 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
I20260619 08:57:41.053879 11685 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:57:41.054593 11685 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:57:41.060912 11685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c83e311c18d491eb35a48825a069e71: Generated new TSK 0
I20260619 08:57:41.061242 11685 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 08:57:41.736879 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60323/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43187
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:57:41.820901 11709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:57:41.821096 11709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:57:41.821125 11709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:57:41.822228 11709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:57:41.822268 11709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:57:41.822301 11709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 08:57:41.823731 11709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:60323/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:57:41.823951 11709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:57:41.824180 11709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:57:41.829823 11717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:57:41.829842 11714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:57:41.830053 11715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:57:41.830339 11709 server_base.cc:1034] running on GCE node
I20260619 08:57:41.830569 11709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:57:41.830859 11709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:57:41.832057 11709 hybrid_clock.cc:648] HybridClock initialized: now 1781859461832038 us; error 33 us; skew 500 ppm
Jun 19 08:57:41 dist-test-slave-h4g5 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:57:41.835223 11709 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 08:57:41.835659 11709 webserver.cc:460] Webserver started at http://127.0.98.1:46411/ using document root <none> and password file <none>
I20260619 08:57:41.835878 11709 fs_manager.cc:359] Metadata directory not provided
I20260619 08:57:41.835928 11709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:57:41.836057 11709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 08:57:41 dist-test-slave-h4g5 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 08:57:41.849926 11709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-0/data/instance:
uuid: "fe48b10b86a846fabb9c0d8ae8d68074"
format_stamp: "Formatted at 2026-06-19 08:57:41 on dist-test-slave-h4g5"
server_key: "f9515074f20996758b5bb7a5dc6ee9c0"
server_key_iv: "8fd40a7c5e0df997627e9cbcb603d613"
server_key_version: "kuduclusterkey@0"
I20260619 08:57:41.850373 11709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fe48b10b86a846fabb9c0d8ae8d68074"
format_stamp: "Formatted at 2026-06-19 08:57:41 on dist-test-slave-h4g5"
server_key: "f9515074f20996758b5bb7a5dc6ee9c0"
server_key_iv: "8fd40a7c5e0df997627e9cbcb603d613"
server_key_version: "kuduclusterkey@0"
I20260619 08:57:41.852201 11709 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260619 08:57:41.870646 11724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:57:41.871052 11709 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:57:41.871168 11709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-0/wal
uuid: "fe48b10b86a846fabb9c0d8ae8d68074"
format_stamp: "Formatted at 2026-06-19 08:57:41 on dist-test-slave-h4g5"
server_key: "f9515074f20996758b5bb7a5dc6ee9c0"
server_key_iv: "8fd40a7c5e0df997627e9cbcb603d613"
server_key_version: "kuduclusterkey@0"
I20260619 08:57:41.871271 11709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:57:41.881286 11709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:57:41.883893 11709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:57:41.884112 11709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:57:41.884402 11709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:57:41.884863 11709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:57:41.884944 11709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:57:41.885039 11709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:57:41.885107 11709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:57:41.891621 11709 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33299
I20260619 08:57:41.892150 11709 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
I20260619 08:57:41.896278 11837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33299 every 8 connection(s)
I20260619 08:57:41.902024 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 11709
I20260619 08:57:41.902122 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-0/wal/instance
I20260619 08:57:41.902318 392 external_mini_cluster.cc:1428] Setting key d37b7a5ed823bc5fa1719d8ff644c3ea
Jun 19 08:57:41 dist-test-slave-h4g5 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:57:41.913955 11841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:57:41.894178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43187 (local address 127.0.98.1:54205)
0619 08:57:41.894346 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 08:57:41.894388 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 08:57:41.895070 (+ 682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:57:41.904461 (+ 9391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:57:41.904467 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:57:41.904729 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:57:41.905026 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:57:41.905034 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:57:41.906080 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:57:41.906081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:57:41.907376 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:57:41.907380 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:57:41.907893 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:57:41.907894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:57:41.908041 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:57:41.908808 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:57:41.908819 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:57:41.910420 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:57:41.912336 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:57:41.912343 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:57:41.912345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:57:41.912621 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:57:41.912858 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:57:41.912859 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:57:41.912859 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:57:41.912988 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:57:41.913277 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:57:41.913284 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:57:41.913457 (+ 173us) client_negotiation.cc:769] Sending connection context
0619 08:57:41.913552 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 08:57:41.913717 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":72,"threads_started":1}
I20260619 08:57:41.914734 11842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:57:41.894437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54205 (local address 127.0.98.62:43187)
0619 08:57:41.904190 (+ 9753us) server_negotiation.cc:206] Beginning negotiation
0619 08:57:41.904193 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:57:41.904204 (+ 11us) server_negotiation.cc:407] Connection header received
0619 08:57:41.904229 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:57:41.904230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:57:41.904242 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:57:41.905125 (+ 883us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:57:41.905165 (+ 40us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:57:41.905964 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:57:41.907476 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:57:41.907797 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:57:41.908525 (+ 728us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:57:41.910516 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:57:41.910517 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:57:41.910518 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:57:41.910525 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:57:41.912242 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:57:41.912722 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:57:41.912723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:57:41.912723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:57:41.912787 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:57:41.913063 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:57:41.913063 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:57:41.913064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:57:41.913197 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:57:41.914494 (+ 1297us) server_negotiation.cc:1035] Waiting for connection context
0619 08:57:41.914537 (+ 43us) server_negotiation.cc:299] Negotiation successful
0619 08:57:41.914621 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9719,"thread_start_us":76,"threads_started":1}
I20260619 08:57:41.915488 11838 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43187
I20260619 08:57:41.915581 11838 heartbeater.cc:464] Registering TS with master...
I20260619 08:57:41.915773 11838 heartbeater.cc:511] Master 127.0.98.62:43187 requested a full tablet report, sending...
I20260619 08:57:41.916360 11628 ts_manager.cc:194] Registered new tserver with Master: fe48b10b86a846fabb9c0d8ae8d68074 (127.0.98.1:33299)
I20260619 08:57:41.917201 11628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54205
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 08:57:41.946681 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60323/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43187
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:57:42.058357 11846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:57:42.058629 11846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:57:42.058669 11846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:57:42.060359 11846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:57:42.060416 11846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:57:42.060454 11846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 08:57:42.062621 11846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:60323/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:57:42.062877 11846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:57:42.063177 11846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:57:42.070093 11856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:57:42.070078 11853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:57:42.070644 11854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:57:42.071214 11846 server_base.cc:1034] running on GCE node
I20260619 08:57:42.071424 11846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:57:42.071683 11846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:57:42.072894 11846 hybrid_clock.cc:648] HybridClock initialized: now 1781859462072859 us; error 47 us; skew 500 ppm
Jun 19 08:57:42 dist-test-slave-h4g5 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:57:42.076661 11846 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 08:57:42.077183 11846 webserver.cc:460] Webserver started at http://127.0.98.2:37365/ using document root <none> and password file <none>
I20260619 08:57:42.077443 11846 fs_manager.cc:359] Metadata directory not provided
I20260619 08:57:42.077504 11846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 08:57:42 dist-test-slave-h4g5 krb5kdc[11152](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781859462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:57:42.077682 11846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 08:57:42 dist-test-slave-h4g5 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 08:57:42.106352 11846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-1/data/instance:
uuid: "93800097993f4354a3375a2ad41e6549"
format_stamp: "Formatted at 2026-06-19 08:57:42 on dist-test-slave-h4g5"
server_key: "0e5c98d26f839227b5af8e3eab2be2f5"
server_key_iv: "6ad6c6a16c688a2ca9adca42b37d4da8"
server_key_version: "kuduclusterkey@0"
I20260619 08:57:42.106902 11846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-1/wal/instance:
uuid: "93800097993f4354a3375a2ad41e6549"
format_stamp: "Formatted at 2026-06-19 08:57:42 on dist-test-slave-h4g5"
server_key: "0e5c98d26f839227b5af8e3eab2be2f5"
server_key_iv: "6ad6c6a16c688a2ca9adca42b37d4da8"
server_key_version: "kuduclusterkey@0"
I20260619 08:57:42.112975 11846 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.000s
I20260619 08:57:42.126498 11863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:57:42.126739 11846 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:57:42.126847 11846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-1/wal
uuid: "93800097993f4354a3375a2ad41e6549"
format_stamp: "Formatted at 2026-06-19 08:57:42 on dist-test-slave-h4g5"
server_key: "0e5c98d26f839227b5af8e3eab2be2f5"
server_key_iv: "6ad6c6a16c688a2ca9adca42b37d4da8"
server_key_version: "kuduclusterkey@0"
I20260619 08:57:42.126940 11846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:57:42.147809 11846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:57:42.150357 11846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:57:42.150578 11846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:57:42.150879 11846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:57:42.151317 11846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:57:42.151412 11846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:57:42.151482 11846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:57:42.151538 11846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:57:42.158648 11846 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39337
I20260619 08:57:42.159157 11846 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
I20260619 08:57:42.160946 11976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39337 every 8 connection(s)
I20260619 08:57:42.162400 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 11846
I20260619 08:57:42.162503 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-1/wal/instance
I20260619 08:57:42.162684 392 external_mini_cluster.cc:1428] Setting key 2476b2f845a9b80d9f85a4148101c8df
Jun 19 08:57:42 dist-test-slave-h4g5 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:57:42.179625 11979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:57:42.161816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43187 (local address 127.0.98.2:53077)
0619 08:57:42.161973 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 08:57:42.162014 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 08:57:42.168792 (+ 6778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:57:42.169206 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:57:42.169224 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:57:42.169475 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:57:42.169762 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:57:42.169769 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:57:42.170812 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:57:42.170813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:57:42.172126 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:57:42.172129 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:57:42.172691 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:57:42.172692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:57:42.172835 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:57:42.173480 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:57:42.173490 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:57:42.175148 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:57:42.177098 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:57:42.177105 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:57:42.177108 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:57:42.177422 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:57:42.178490 (+ 1068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:57:42.178491 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:57:42.178492 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:57:42.178632 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:57:42.178909 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:57:42.178915 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:57:42.179098 (+ 183us) client_negotiation.cc:769] Sending connection context
0619 08:57:42.179222 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 08:57:42.179384 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260619 08:57:42.180176 11842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:57:42.161834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53077 (local address 127.0.98.62:43187)
0619 08:57:42.161895 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 08:57:42.161897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:57:42.162095 (+ 198us) server_negotiation.cc:407] Connection header received
0619 08:57:42.168951 (+ 6856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:57:42.168952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:57:42.168969 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:57:42.168999 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:57:42.169879 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:57:42.170696 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:57:42.172236 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:57:42.172553 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:57:42.172596 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:57:42.175281 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:57:42.175283 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:57:42.175284 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:57:42.175293 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:57:42.177005 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:57:42.178357 (+ 1352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:57:42.178358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:57:42.178359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:57:42.178427 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:57:42.178694 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:57:42.178695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:57:42.178696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:57:42.178831 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:57:42.178886 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 08:57:42.179963 (+ 1077us) server_negotiation.cc:299] Negotiation successful
0619 08:57:42.180057 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 08:57:42.180850 11977 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43187
I20260619 08:57:42.180953 11977 heartbeater.cc:464] Registering TS with master...
I20260619 08:57:42.181139 11977 heartbeater.cc:511] Master 127.0.98.62:43187 requested a full tablet report, sending...
I20260619 08:57:42.181488 11628 ts_manager.cc:194] Registered new tserver with Master: 93800097993f4354a3375a2ad41e6549 (127.0.98.2:39337)
I20260619 08:57:42.181857 11628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53077
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 08:57:42.211024 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60323/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43187
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:57:42.334291 11983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:57:42.334672 11983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:57:42.334755 11983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:57:42.336786 11983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:57:42.337003 11983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:57:42.337086 11983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 08:57:42.339228 11983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:60323/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:57:42.339685 11983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:57:42.339988 11983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:57:42.347000 11988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:57:42.347069 11983 server_base.cc:1034] running on GCE node
W20260619 08:57:42.348222 11989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:57:42.351019 11991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:57:42.351656 11983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:57:42.351962 11983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:57:42.359635 11983 hybrid_clock.cc:648] HybridClock initialized: now 1781859462359607 us; error 41 us; skew 500 ppm
Jun 19 08:57:42 dist-test-slave-h4g5 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:57:42.363224 11983 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 08:57:42.363798 11983 webserver.cc:460] Webserver started at http://127.0.98.3:41903/ using document root <none> and password file <none>
I20260619 08:57:42.364070 11983 fs_manager.cc:359] Metadata directory not provided
I20260619 08:57:42.364173 11983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:57:42.364354 11983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 08:57:42 dist-test-slave-h4g5 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 08:57:42.383821 11983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-2/data/instance:
uuid: "b44fc2f6fca040faa2b52c3a6dde0a4c"
format_stamp: "Formatted at 2026-06-19 08:57:42 on dist-test-slave-h4g5"
server_key: "33f234b2776d03ac1aaaf11ff3511d89"
server_key_iv: "f275e9731bd5c745dabee81d47bcfb53"
server_key_version: "kuduclusterkey@0"
I20260619 08:57:42.384279 11983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b44fc2f6fca040faa2b52c3a6dde0a4c"
format_stamp: "Formatted at 2026-06-19 08:57:42 on dist-test-slave-h4g5"
server_key: "33f234b2776d03ac1aaaf11ff3511d89"
server_key_iv: "f275e9731bd5c745dabee81d47bcfb53"
server_key_version: "kuduclusterkey@0"
I20260619 08:57:42.388497 11983 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.002s
I20260619 08:57:42.404247 11998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:57:42.404537 11983 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260619 08:57:42.404659 11983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-2/wal
uuid: "b44fc2f6fca040faa2b52c3a6dde0a4c"
format_stamp: "Formatted at 2026-06-19 08:57:42 on dist-test-slave-h4g5"
server_key: "33f234b2776d03ac1aaaf11ff3511d89"
server_key_iv: "f275e9731bd5c745dabee81d47bcfb53"
server_key_version: "kuduclusterkey@0"
I20260619 08:57:42.404760 11983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:57:42.417627 11983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:57:42.420024 11983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:57:42.420212 11983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:57:42.420480 11983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:57:42.420843 11983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:57:42.420893 11983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:57:42.420929 11983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:57:42.420951 11983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:57:42.429911 11983 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43253
I20260619 08:57:42.430323 11983 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
I20260619 08:57:42.434270 12111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43253 every 8 connection(s)
Jun 19 08:57:42 dist-test-slave-h4g5 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:57:42.440454 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 11983
I20260619 08:57:42.440548 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-2/wal/instance
I20260619 08:57:42.440721 392 external_mini_cluster.cc:1428] Setting key 19d81e985d4729863080db35d97b37a3
I20260619 08:57:42.444235 11842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:57:42.434918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56827 (local address 127.0.98.62:43187)
0619 08:57:42.434963 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 08:57:42.434965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:57:42.435192 (+ 227us) server_negotiation.cc:407] Connection header received
0619 08:57:42.435705 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:57:42.435707 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:57:42.435718 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:57:42.435765 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:57:42.436399 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:57:42.436893 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:57:42.438293 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:57:42.438484 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:57:42.438540 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:57:42.441251 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:57:42.441253 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:57:42.441253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:57:42.441260 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:57:42.442637 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:57:42.443120 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:57:42.443122 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:57:42.443122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:57:42.443180 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:57:42.443451 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:57:42.443452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:57:42.443452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:57:42.443594 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:57:42.443637 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 08:57:42.444025 (+ 388us) server_negotiation.cc:299] Negotiation successful
0619 08:57:42.444139 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 08:57:42.444765 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:57:42.434917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43187 (local address 127.0.98.3:56827)
0619 08:57:42.435070 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 08:57:42.435112 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 08:57:42.435607 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:57:42.435849 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:57:42.435853 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:57:42.436068 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:57:42.436308 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:57:42.436313 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:57:42.436976 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:57:42.436977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:57:42.438213 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:57:42.438216 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:57:42.438563 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:57:42.438564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:57:42.438711 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:57:42.439282 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:57:42.439292 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:57:42.441095 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:57:42.442722 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:57:42.442729 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:57:42.442731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:57:42.443015 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:57:42.443246 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:57:42.443247 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:57:42.443247 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:57:42.443376 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:57:42.443685 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:57:42.443690 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:57:42.443866 (+ 176us) client_negotiation.cc:769] Sending connection context
0619 08:57:42.444366 (+ 500us) client_negotiation.cc:240] Negotiation successful
0619 08:57:42.444525 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260619 08:57:42.445163 12112 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43187
I20260619 08:57:42.445252 12112 heartbeater.cc:464] Registering TS with master...
I20260619 08:57:42.445449 12112 heartbeater.cc:511] Master 127.0.98.62:43187 requested a full tablet report, sending...
I20260619 08:57:42.445775 11628 ts_manager.cc:194] Registered new tserver with Master: b44fc2f6fca040faa2b52c3a6dde0a4c (127.0.98.3:43253)
I20260619 08:57:42.446244 11628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56827
I20260619 08:57:42.452723 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:57:42.467626 11842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:57:42.454417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58452 (local address 127.0.98.62:43187)
0619 08:57:42.454460 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 08:57:42.454462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:57:42.454716 (+ 254us) server_negotiation.cc:407] Connection header received
0619 08:57:42.454807 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:57:42.454808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:57:42.454818 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:57:42.454846 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:57:42.455596 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:57:42.456381 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:57:42.458552 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:57:42.458834 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:57:42.458874 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:57:42.460540 (+ 1666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:57:42.460541 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:57:42.460542 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:57:42.460549 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:57:42.466119 (+ 5570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:57:42.466583 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:57:42.466585 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:57:42.466586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:57:42.466654 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:57:42.466868 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:57:42.466869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:57:42.466870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:57:42.467005 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:57:42.467095 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 08:57:42.467442 (+ 347us) server_negotiation.cc:299] Negotiation successful
0619 08:57:42.467528 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 08:57:42.469913 11628 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58452:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:57:42.472204 11628 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:57:42.500259 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:57:42.489685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43253 (local address 127.0.0.1:47244)
0619 08:57:42.490990 (+ 1305us) negotiation.cc:107] Waiting for socket to connect
0619 08:57:42.491033 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 08:57:42.491096 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:57:42.491438 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:57:42.491441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:57:42.491444 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:57:42.491528 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:57:42.491530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:57:42.496518 (+ 4988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:57:42.496519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:57:42.497906 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:57:42.497908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:57:42.499918 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:57:42.499920 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:57:42.500018 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:57:42.500031 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 08:57:42.500114 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 08:57:42.500170 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1278,"thread_start_us":77,"threads_started":1}
I20260619 08:57:42.500655 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:57:42.490392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33299 (local address 127.0.0.1:54920)
0619 08:57:42.492859 (+ 2467us) negotiation.cc:107] Waiting for socket to connect
0619 08:57:42.492866 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 08:57:42.492897 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:57:42.493126 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:57:42.493127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:57:42.493129 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:57:42.493175 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:57:42.493177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:57:42.494379 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:57:42.494380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:57:42.495995 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:57:42.495998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:57:42.500433 (+ 4435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:57:42.500434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:57:42.500496 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:57:42.500499 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:57:42.500538 (+ 39us) client_negotiation.cc:240] Negotiation successful
0619 08:57:42.500576 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2456,"thread_start_us":62,"threads_started":1}
I20260619 08:57:42.501188 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:57:42.490608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54920 (local address 127.0.98.1:33299)
0619 08:57:42.491210 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0619 08:57:42.491215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:57:42.492948 (+ 1733us) server_negotiation.cc:407] Connection header received
0619 08:57:42.492985 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:57:42.492988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:57:42.493003 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:57:42.493063 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:57:42.493236 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:57:42.494289 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:57:42.497991 (+ 3702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:57:42.498798 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:57:42.498866 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:57:42.498926 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 08:57:42.501060 (+ 2134us) server_negotiation.cc:299] Negotiation successful
0619 08:57:42.501115 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":44,"threads_started":1}
I20260619 08:57:42.501848 11772 tablet_service.cc:1467] Processing CreateTablet for tablet 8bcf9d447159404ba10c5f0f700d4c00 (DEFAULT_TABLE table=test-table [id=b89ecfc63ec141fa8149e2d50815a9f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:57:42.502206 11772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bcf9d447159404ba10c5f0f700d4c00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:57:42.506197 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:57:42.490070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47244 (local address 127.0.98.3:43253)
0619 08:57:42.490930 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0619 08:57:42.490935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:57:42.491235 (+ 300us) server_negotiation.cc:407] Connection header received
0619 08:57:42.491275 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:57:42.491278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:57:42.491293 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:57:42.491352 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:57:42.491624 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:57:42.496436 (+ 4812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:57:42.498981 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:57:42.499754 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:57:42.499821 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:57:42.499885 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 08:57:42.506065 (+ 6180us) server_negotiation.cc:299] Negotiation successful
0619 08:57:42.506118 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":824,"thread_start_us":101,"threads_started":1}
I20260619 08:57:42.506834 12046 tablet_service.cc:1467] Processing CreateTablet for tablet 8bcf9d447159404ba10c5f0f700d4c00 (DEFAULT_TABLE table=test-table [id=b89ecfc63ec141fa8149e2d50815a9f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:57:42.507159 12046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bcf9d447159404ba10c5f0f700d4c00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:57:42.508598 12125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:57:42.489926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39337 (local address 127.0.0.1:52308)
0619 08:57:42.492750 (+ 2824us) negotiation.cc:107] Waiting for socket to connect
0619 08:57:42.492759 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:57:42.492798 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:57:42.504996 (+ 12198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:57:42.504997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:57:42.505000 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:57:42.505052 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:57:42.505054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:57:42.506260 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:57:42.506262 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:57:42.507362 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:57:42.507364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:57:42.508366 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:57:42.508367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:57:42.508431 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:57:42.508437 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 08:57:42.508485 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 08:57:42.508526 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2807,"thread_start_us":75,"threads_started":1}
I20260619 08:57:42.508822 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:57:42.490036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52308 (local address 127.0.98.2:39337)
0619 08:57:42.504213 (+ 14177us) server_negotiation.cc:206] Beginning negotiation
0619 08:57:42.504219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 08:57:42.504230 (+ 11us) server_negotiation.cc:407] Connection header received
0619 08:57:42.504268 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:57:42.504270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:57:42.504287 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:57:42.504897 (+ 610us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:57:42.505145 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:57:42.506167 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:57:42.507478 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:57:42.508197 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:57:42.508264 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:57:42.508331 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 08:57:42.508703 (+ 372us) server_negotiation.cc:299] Negotiation successful
0619 08:57:42.508745 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14135,"thread_start_us":83,"threads_started":1}
I20260619 08:57:42.510123 11909 tablet_service.cc:1467] Processing CreateTablet for tablet 8bcf9d447159404ba10c5f0f700d4c00 (DEFAULT_TABLE table=test-table [id=b89ecfc63ec141fa8149e2d50815a9f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:57:42.510456 11909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bcf9d447159404ba10c5f0f700d4c00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:57:42.515694 12132 tablet_bootstrap.cc:492] T 8bcf9d447159404ba10c5f0f700d4c00 P b44fc2f6fca040faa2b52c3a6dde0a4c: Bootstrap starting.
I20260619 08:57:42.516220 12133 tablet_bootstrap.cc:492] T 8bcf9d447159404ba10c5f0f700d4c00 P 93800097993f4354a3375a2ad41e6549: Bootstrap starting.
I20260619 08:57:42.517437 12132 tablet_bootstrap.cc:654] T 8bcf9d447159404ba10c5f0f700d4c00 P b44fc2f6fca040faa2b52c3a6dde0a4c: Neither blocks nor log segments found. Creating new log.
I20260619 08:57:42.517858 12132 log.cc:824] T 8bcf9d447159404ba10c5f0f700d4c00 P b44fc2f6fca040faa2b52c3a6dde0a4c: Log is configured to *not* fsync() on all Append() calls
I20260619 08:57:42.518390 12133 tablet_bootstrap.cc:654] T 8bcf9d447159404ba10c5f0f700d4c00 P 93800097993f4354a3375a2ad41e6549: Neither blocks nor log segments found. Creating new log.
I20260619 08:57:42.518715 12133 log.cc:824] T 8bcf9d447159404ba10c5f0f700d4c00 P 93800097993f4354a3375a2ad41e6549: Log is configured to *not* fsync() on all Append() calls
I20260619 08:57:42.519085 12131 tablet_bootstrap.cc:492] T 8bcf9d447159404ba10c5f0f700d4c00 P fe48b10b86a846fabb9c0d8ae8d68074: Bootstrap starting.
I20260619 08:57:42.520727 12131 tablet_bootstrap.cc:654] T 8bcf9d447159404ba10c5f0f700d4c00 P fe48b10b86a846fabb9c0d8ae8d68074: Neither blocks nor log segments found. Creating new log.
I20260619 08:57:42.520998 12131 log.cc:824] T 8bcf9d447159404ba10c5f0f700d4c00 P fe48b10b86a846fabb9c0d8ae8d68074: Log is configured to *not* fsync() on all Append() calls
I20260619 08:57:42.521965 12132 tablet_bootstrap.cc:492] T 8bcf9d447159404ba10c5f0f700d4c00 P b44fc2f6fca040faa2b52c3a6dde0a4c: No bootstrap required, opened a new log
I20260619 08:57:42.522040 12132 ts_tablet_manager.cc:1397] T 8bcf9d447159404ba10c5f0f700d4c00 P b44fc2f6fca040faa2b52c3a6dde0a4c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260619 08:57:42.523928 12132 raft_consensus.cc:348] T 8bcf9d447159404ba10c5f0f700d4c00 P b44fc2f6fca040faa2b52c3a6dde0a4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44fc2f6fca040faa2b52c3a6dde0a4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43253 } } peers { permanent_uuid: "fe48b10b86a846fabb9c0d8ae8d68074" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33299 } } peers { permanent_uuid: "93800097993f4354a3375a2ad41e6549" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39337 } }
I20260619 08:57:42.524142 12132 raft_consensus.cc:374] T 8bcf9d447159404ba10c5f0f700d4c00 P b44fc2f6fca040faa2b52c3a6dde0a4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:57:42.524186 12132 raft_consensus.cc:729] T 8bcf9d447159404ba10c5f0f700d4c00 P b44fc2f6fca040faa2b52c3a6dde0a4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b44fc2f6fca040faa2b52c3a6dde0a4c, State: Initialized, Role: FOLLOWER
I20260619 08:57:42.524322 12132 consensus_queue.cc:260] T 8bcf9d447159404ba10c5f0f700d4c00 P b44fc2f6fca040faa2b52c3a6dde0a4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44fc2f6fca040faa2b52c3a6dde0a4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43253 } } peers { permanent_uuid: "fe48b10b86a846fabb9c0d8ae8d68074" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33299 } } peers { permanent_uuid: "93800097993f4354a3375a2ad41e6549" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39337 } }
I20260619 08:57:42.524578 12132 ts_tablet_manager.cc:1428] T 8bcf9d447159404ba10c5f0f700d4c00 P b44fc2f6fca040faa2b52c3a6dde0a4c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 08:57:42.524701 12112 heartbeater.cc:503] Master 127.0.98.62:43187 was elected leader, sending a full tablet report...
I20260619 08:57:42.525978 12133 tablet_bootstrap.cc:492] T 8bcf9d447159404ba10c5f0f700d4c00 P 93800097993f4354a3375a2ad41e6549: No bootstrap required, opened a new log
I20260619 08:57:42.526062 12133 ts_tablet_manager.cc:1397] T 8bcf9d447159404ba10c5f0f700d4c00 P 93800097993f4354a3375a2ad41e6549: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260619 08:57:42.528429 12133 raft_consensus.cc:348] T 8bcf9d447159404ba10c5f0f700d4c00 P 93800097993f4354a3375a2ad41e6549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44fc2f6fca040faa2b52c3a6dde0a4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43253 } } peers { permanent_uuid: "fe48b10b86a846fabb9c0d8ae8d68074" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33299 } } peers { permanent_uuid: "93800097993f4354a3375a2ad41e6549" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39337 } }
I20260619 08:57:42.528621 12133 raft_consensus.cc:374] T 8bcf9d447159404ba10c5f0f700d4c00 P 93800097993f4354a3375a2ad41e6549 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:57:42.528658 12133 raft_consensus.cc:729] T 8bcf9d447159404ba10c5f0f700d4c00 P 93800097993f4354a3375a2ad41e6549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93800097993f4354a3375a2ad41e6549, State: Initialized, Role: FOLLOWER
I20260619 08:57:42.528785 12133 consensus_queue.cc:260] T 8bcf9d447159404ba10c5f0f700d4c00 P 93800097993f4354a3375a2ad41e6549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44fc2f6fca040faa2b52c3a6dde0a4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43253 } } peers { permanent_uuid: "fe48b10b86a846fabb9c0d8ae8d68074" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33299 } } peers { permanent_uuid: "93800097993f4354a3375a2ad41e6549" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39337 } }
I20260619 08:57:42.529055 12133 ts_tablet_manager.cc:1428] T 8bcf9d447159404ba10c5f0f700d4c00 P 93800097993f4354a3375a2ad41e6549: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:57:42.529165 11977 heartbeater.cc:503] Master 127.0.98.62:43187 was elected leader, sending a full tablet report...
I20260619 08:57:42.530411 12131 tablet_bootstrap.cc:492] T 8bcf9d447159404ba10c5f0f700d4c00 P fe48b10b86a846fabb9c0d8ae8d68074: No bootstrap required, opened a new log
I20260619 08:57:42.530529 12131 ts_tablet_manager.cc:1397] T 8bcf9d447159404ba10c5f0f700d4c00 P fe48b10b86a846fabb9c0d8ae8d68074: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260619 08:57:42.533223 12131 raft_consensus.cc:348] T 8bcf9d447159404ba10c5f0f700d4c00 P fe48b10b86a846fabb9c0d8ae8d68074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44fc2f6fca040faa2b52c3a6dde0a4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43253 } } peers { permanent_uuid: "fe48b10b86a846fabb9c0d8ae8d68074" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33299 } } peers { permanent_uuid: "93800097993f4354a3375a2ad41e6549" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39337 } }
I20260619 08:57:42.533437 12131 raft_consensus.cc:374] T 8bcf9d447159404ba10c5f0f700d4c00 P fe48b10b86a846fabb9c0d8ae8d68074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:57:42.533491 12131 raft_consensus.cc:729] T 8bcf9d447159404ba10c5f0f700d4c00 P fe48b10b86a846fabb9c0d8ae8d68074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe48b10b86a846fabb9c0d8ae8d68074, State: Initialized, Role: FOLLOWER
I20260619 08:57:42.533749 12131 consensus_queue.cc:260] T 8bcf9d447159404ba10c5f0f700d4c00 P fe48b10b86a846fabb9c0d8ae8d68074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44fc2f6fca040faa2b52c3a6dde0a4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43253 } } peers { permanent_uuid: "fe48b10b86a846fabb9c0d8ae8d68074" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33299 } } peers { permanent_uuid: "93800097993f4354a3375a2ad41e6549" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39337 } }
I20260619 08:57:42.534010 12131 ts_tablet_manager.cc:1428] T 8bcf9d447159404ba10c5f0f700d4c00 P fe48b10b86a846fabb9c0d8ae8d68074: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:57:42.534118 11838 heartbeater.cc:503] Master 127.0.98.62:43187 was elected leader, sending a full tablet report...
I20260619 08:57:42.554201 12138 raft_consensus.cc:482] T 8bcf9d447159404ba10c5f0f700d4c00 P 93800097993f4354a3375a2ad41e6549 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:57:42.554431 12138 raft_consensus.cc:504] T 8bcf9d447159404ba10c5f0f700d4c00 P 93800097993f4354a3375a2ad41e6549 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44fc2f6fca040faa2b52c3a6dde0a4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43253 } } peers { permanent_uuid: "fe48b10b86a846fabb9c0d8ae8d68074" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33299 } } peers { permanent_uuid: "93800097993f4354a3375a2ad41e6549" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39337 } }
I20260619 08:57:42.558516 11979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:57:42.554984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43253 (local address 127.0.98.2:38907)
0619 08:57:42.555029 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 08:57:42.555042 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 08:57:42.555101 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:57:42.555278 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:57:42.555279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:57:42.555281 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:57:42.555349 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:57:42.555350 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:57:42.556339 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:57:42.556340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:57:42.557530 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:57:42.557533 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:57:42.558279 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:57:42.558280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:57:42.558354 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:57:42.558358 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:57:42.558399 (+ 41us) client_negotiation.cc:240] Negotiation successful
0619 08:57:42.558438 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260619 08:57:42.558800 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:57:42.555100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38907 (local address 127.0.98.3:43253)
0619 08:57:42.555140 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 08:57:42.555142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:57:42.555148 (+ 6us) server_negotiation.cc:407] Connection header received
0619 08:57:42.555174 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:57:42.555175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:57:42.555182 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:57:42.555224 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:57:42.555416 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:57:42.556185 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:57:42.557636 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:57:42.558173 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:57:42.558238 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:57:42.558249 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 08:57:42.558696 (+ 447us) server_negotiation.cc:299] Negotiation successful
0619 08:57:42.558736 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 08:57:42.559051 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bcf9d447159404ba10c5f0f700d4c00" candidate_uuid: "93800097993f4354a3375a2ad41e6549" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b44fc2f6fca040faa2b52c3a6dde0a4c" is_pre_election: true
I20260619 08:57:42.559221 12047 raft_consensus.cc:2454] T 8bcf9d447159404ba10c5f0f700d4c00 P b44fc2f6fca040faa2b52c3a6dde0a4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93800097993f4354a3375a2ad41e6549 in term 0.
I20260619 08:57:42.559465 11866 leader_election.cc:302] T 8bcf9d447159404ba10c5f0f700d4c00 P 93800097993f4354a3375a2ad41e6549 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93800097993f4354a3375a2ad41e6549, b44fc2f6fca040faa2b52c3a6dde0a4c; no voters:
I20260619 08:57:42.563344 11979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:57:42.559854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33299 (local address 127.0.98.2:56657)
0619 08:57:42.560012 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 08:57:42.560021 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:57:42.560057 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:57:42.560255 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:57:42.560257 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:57:42.560258 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:57:42.560291 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:57:42.560292 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:57:42.561158 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:57:42.561159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:57:42.562331 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:57:42.562333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:57:42.563128 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:57:42.563129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:57:42.563190 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:57:42.563193 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:57:42.563236 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 08:57:42.563271 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260619 08:57:42.563614 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:57:42.559949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56657 (local address 127.0.98.1:33299)
0619 08:57:42.559987 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 08:57:42.559989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:57:42.560111 (+ 122us) server_negotiation.cc:407] Connection header received
0619 08:57:42.560172 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:57:42.560173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:57:42.560182 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:57:42.560218 (+ 36us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:57:42.560345 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:57:42.561070 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:57:42.562426 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:57:42.563020 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:57:42.563086 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:57:42.563097 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 08:57:42.563496 (+ 399us) server_negotiation.cc:299] Negotiation successful
0619 08:57:42.563540 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 08:57:42.563868 11792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bcf9d447159404ba10c5f0f700d4c00" candidate_uuid: "93800097993f4354a3375a2ad41e6549" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe48b10b86a846fabb9c0d8ae8d68074" is_pre_election: true
I20260619 08:57:42.564023 11792 raft_consensus.cc:2454] T 8bcf9d447159404ba10c5f0f700d4c00 P fe48b10b86a846fabb9c0d8ae8d68074 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93800097993f4354a3375a2ad41e6549 in term 0.
I20260619 08:57:42.564254 12138 leader_election.cc:288] T 8bcf9d447159404ba10c5f0f700d4c00 P 93800097993f4354a3375a2ad41e6549 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b44fc2f6fca040faa2b52c3a6dde0a4c (127.0.98.3:43253), fe48b10b86a846fabb9c0d8ae8d68074 (127.0.98.1:33299)
I20260619 08:57:42.564370 12138 raft_consensus.cc:2781] T 8bcf9d447159404ba10c5f0f700d4c00 P 93800097993f4354a3375a2ad41e6549 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:57:42.564417 12138 raft_consensus.cc:482] T 8bcf9d447159404ba10c5f0f700d4c00 P 93800097993f4354a3375a2ad41e6549 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:57:42.564441 12138 raft_consensus.cc:3037] T 8bcf9d447159404ba10c5f0f700d4c00 P 93800097993f4354a3375a2ad41e6549 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:57:42.565737 12138 raft_consensus.cc:504] T 8bcf9d447159404ba10c5f0f700d4c00 P 93800097993f4354a3375a2ad41e6549 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44fc2f6fca040faa2b52c3a6dde0a4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43253 } } peers { permanent_uuid: "fe48b10b86a846fabb9c0d8ae8d68074" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33299 } } peers { permanent_uuid: "93800097993f4354a3375a2ad41e6549" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39337 } }
I20260619 08:57:42.566010 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bcf9d447159404ba10c5f0f700d4c00" candidate_uuid: "93800097993f4354a3375a2ad41e6549" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b44fc2f6fca040faa2b52c3a6dde0a4c"
I20260619 08:57:42.566099 12047 raft_consensus.cc:3037] T 8bcf9d447159404ba10c5f0f700d4c00 P b44fc2f6fca040faa2b52c3a6dde0a4c [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:57:42.566951 12047 raft_consensus.cc:2454] T 8bcf9d447159404ba10c5f0f700d4c00 P b44fc2f6fca040faa2b52c3a6dde0a4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93800097993f4354a3375a2ad41e6549 in term 1.
I20260619 08:57:42.567152 11866 leader_election.cc:302] T 8bcf9d447159404ba10c5f0f700d4c00 P 93800097993f4354a3375a2ad41e6549 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93800097993f4354a3375a2ad41e6549, b44fc2f6fca040faa2b52c3a6dde0a4c; no voters:
I20260619 08:57:42.567351 11792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bcf9d447159404ba10c5f0f700d4c00" candidate_uuid: "93800097993f4354a3375a2ad41e6549" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe48b10b86a846fabb9c0d8ae8d68074"
I20260619 08:57:42.567431 11792 raft_consensus.cc:3037] T 8bcf9d447159404ba10c5f0f700d4c00 P fe48b10b86a846fabb9c0d8ae8d68074 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:57:42.568396 11792 raft_consensus.cc:2454] T 8bcf9d447159404ba10c5f0f700d4c00 P fe48b10b86a846fabb9c0d8ae8d68074 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93800097993f4354a3375a2ad41e6549 in term 1.
I20260619 08:57:42.568634 12138 leader_election.cc:288] T 8bcf9d447159404ba10c5f0f700d4c00 P 93800097993f4354a3375a2ad41e6549 [CANDIDATE]: Term 1 election: Requested vote from peers b44fc2f6fca040faa2b52c3a6dde0a4c (127.0.98.3:43253), fe48b10b86a846fabb9c0d8ae8d68074 (127.0.98.1:33299)
I20260619 08:57:42.568706 12138 raft_consensus.cc:2781] T 8bcf9d447159404ba10c5f0f700d4c00 P 93800097993f4354a3375a2ad41e6549 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:57:42.568747 12138 raft_consensus.cc:686] T 8bcf9d447159404ba10c5f0f700d4c00 P 93800097993f4354a3375a2ad41e6549 [term 1 LEADER]: Becoming Leader. State: Replica: 93800097993f4354a3375a2ad41e6549, State: Running, Role: LEADER
I20260619 08:57:42.568832 12138 consensus_queue.cc:237] T 8bcf9d447159404ba10c5f0f700d4c00 P 93800097993f4354a3375a2ad41e6549 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44fc2f6fca040faa2b52c3a6dde0a4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43253 } } peers { permanent_uuid: "fe48b10b86a846fabb9c0d8ae8d68074" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33299 } } peers { permanent_uuid: "93800097993f4354a3375a2ad41e6549" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39337 } }
I20260619 08:57:42.569766 11628 catalog_manager.cc:5495] T 8bcf9d447159404ba10c5f0f700d4c00 P 93800097993f4354a3375a2ad41e6549 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93800097993f4354a3375a2ad41e6549 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "93800097993f4354a3375a2ad41e6549" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b44fc2f6fca040faa2b52c3a6dde0a4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe48b10b86a846fabb9c0d8ae8d68074" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93800097993f4354a3375a2ad41e6549" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39337 } health_report { overall_health: HEALTHY } } }
I20260619 08:57:42.588874 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:57:42.585618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52320 (local address 127.0.98.2:39337)
0619 08:57:42.585666 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 08:57:42.585668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:57:42.585676 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:57:42.585714 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:57:42.585715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:57:42.585736 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:57:42.585804 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:57:42.585988 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:57:42.586674 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:57:42.587757 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:57:42.588055 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:57:42.588156 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:57:42.588345 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:57:42.588480 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:57:42.588544 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 08:57:42.588745 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 08:57:42.588778 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 08:57:42.591374 11792 raft_consensus.cc:1261] T 8bcf9d447159404ba10c5f0f700d4c00 P fe48b10b86a846fabb9c0d8ae8d68074 [term 1 FOLLOWER]: Refusing update from remote peer 93800097993f4354a3375a2ad41e6549: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:57:42.591715 12140 consensus_queue.cc:1035] T 8bcf9d447159404ba10c5f0f700d4c00 P 93800097993f4354a3375a2ad41e6549 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe48b10b86a846fabb9c0d8ae8d68074" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:57:42.593779 12047 raft_consensus.cc:1261] T 8bcf9d447159404ba10c5f0f700d4c00 P b44fc2f6fca040faa2b52c3a6dde0a4c [term 1 FOLLOWER]: Refusing update from remote peer 93800097993f4354a3375a2ad41e6549: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:57:42.594043 12140 consensus_queue.cc:1035] T 8bcf9d447159404ba10c5f0f700d4c00 P 93800097993f4354a3375a2ad41e6549 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b44fc2f6fca040faa2b52c3a6dde0a4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:57:42.603266 12143 mvcc.cc:205] Tried to move back new op lower bound from 7298496358771367936 to 7298496358682398720. Current Snapshot: MvccSnapshot[applied={T|T < 7298496358771367936}]
I20260619 08:57:42.603489 12146 mvcc.cc:205] Tried to move back new op lower bound from 7298496358771367936 to 7298496358682398720. Current Snapshot: MvccSnapshot[applied={T|T < 7298496358771367936}]
I20260619 08:57:42.603657 12144 mvcc.cc:205] Tried to move back new op lower bound from 7298496358771367936 to 7298496358682398720. Current Snapshot: MvccSnapshot[applied={T|T < 7298496358771367936}]
I20260619 08:57:42.608023 11628 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:57:42.608155 11628 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:57:42.608868 11628 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c83e311c18d491eb35a48825a069e71: Sending DeleteTablet for 3 replicas of tablet 8bcf9d447159404ba10c5f0f700d4c00
I20260619 08:57:42.609261 12046 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bcf9d447159404ba10c5f0f700d4c00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:57:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47244
I20260619 08:57:42.609344 11772 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bcf9d447159404ba10c5f0f700d4c00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:57:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54920
I20260619 08:57:42.609609 11909 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bcf9d447159404ba10c5f0f700d4c00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:57:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52308
I20260619 08:57:42.610103 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 11709
I20260619 08:57:42.616219 12154 tablet_replica.cc:330] T 8bcf9d447159404ba10c5f0f700d4c00 P 93800097993f4354a3375a2ad41e6549: stopping tablet replica
I20260619 08:57:42.616336 12154 raft_consensus.cc:2229] T 8bcf9d447159404ba10c5f0f700d4c00 P 93800097993f4354a3375a2ad41e6549 [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:57:42.616472 12154 raft_consensus.cc:2258] T 8bcf9d447159404ba10c5f0f700d4c00 P 93800097993f4354a3375a2ad41e6549 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:57:42.617478 12154 ts_tablet_manager.cc:1905] T 8bcf9d447159404ba10c5f0f700d4c00 P 93800097993f4354a3375a2ad41e6549: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:57:42.624205 12152 tablet_replica.cc:330] T 8bcf9d447159404ba10c5f0f700d4c00 P b44fc2f6fca040faa2b52c3a6dde0a4c: stopping tablet replica
I20260619 08:57:42.624347 12152 raft_consensus.cc:2229] T 8bcf9d447159404ba10c5f0f700d4c00 P b44fc2f6fca040faa2b52c3a6dde0a4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:57:42.624473 12152 raft_consensus.cc:2258] T 8bcf9d447159404ba10c5f0f700d4c00 P b44fc2f6fca040faa2b52c3a6dde0a4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:57:42.624493 12154 ts_tablet_manager.cc:1918] T 8bcf9d447159404ba10c5f0f700d4c00 P 93800097993f4354a3375a2ad41e6549: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:57:42.624643 12154 log.cc:1196] T 8bcf9d447159404ba10c5f0f700d4c00 P 93800097993f4354a3375a2ad41e6549: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-1/wal/wals/8bcf9d447159404ba10c5f0f700d4c00
W20260619 08:57:42.624732 11615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33299 (error 108)
I20260619 08:57:42.625041 12154 ts_tablet_manager.cc:1939] T 8bcf9d447159404ba10c5f0f700d4c00 P 93800097993f4354a3375a2ad41e6549: Deleting consensus metadata
I20260619 08:57:42.625496 11612 catalog_manager.cc:4841] TS 93800097993f4354a3375a2ad41e6549 (127.0.98.2:39337): tablet 8bcf9d447159404ba10c5f0f700d4c00 (table test-table [id=b89ecfc63ec141fa8149e2d50815a9f4]) successfully deleted
I20260619 08:57:42.625685 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 11846
I20260619 08:57:42.627365 12152 ts_tablet_manager.cc:1905] T 8bcf9d447159404ba10c5f0f700d4c00 P b44fc2f6fca040faa2b52c3a6dde0a4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:57:42.632292 12152 ts_tablet_manager.cc:1918] T 8bcf9d447159404ba10c5f0f700d4c00 P b44fc2f6fca040faa2b52c3a6dde0a4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:57:42.632375 12152 log.cc:1196] T 8bcf9d447159404ba10c5f0f700d4c00 P b44fc2f6fca040faa2b52c3a6dde0a4c: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859361540516-392-0/minicluster-data/ts-2/wal/wals/8bcf9d447159404ba10c5f0f700d4c00
I20260619 08:57:42.632618 12152 ts_tablet_manager.cc:1939] T 8bcf9d447159404ba10c5f0f700d4c00 P b44fc2f6fca040faa2b52c3a6dde0a4c: Deleting consensus metadata
I20260619 08:57:42.633073 11614 catalog_manager.cc:4841] TS b44fc2f6fca040faa2b52c3a6dde0a4c (127.0.98.3:43253): tablet 8bcf9d447159404ba10c5f0f700d4c00 (table test-table [id=b89ecfc63ec141fa8149e2d50815a9f4]) successfully deleted
Jun 19 08:57:42 dist-test-slave-h4g5 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 08:57:42.638957 12125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:57:42.638837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33299 (local address 127.0.0.1:54932)
0619 08:57:42.638883 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 08:57:42.638917 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260619 08:57:42.639091 11615 catalog_manager.cc:4584] TS fe48b10b86a846fabb9c0d8ae8d68074 (127.0.98.1:33299): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8bcf9d447159404ba10c5f0f700d4c00: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33299: connect: Connection refused (error 111)
I20260619 08:57:42.639497 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 11983
I20260619 08:57:42.646209 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 11596
I20260619 08:57:42.650871 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 08:57:42.995663 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 08:57:42.995782 392 mini_ranger.cc:66] Stopping Ranger...
I20260619 08:57:43.330673 392 mini_ranger.cc:68] Stopped Ranger
2026-06-19 08:57:43.330 UTC [11182] LOG: received smart shutdown request
2026-06-19 08:57:43.334 UTC [11182] LOG: background worker "logical replication launcher" (PID 11191) exited with exit code 1
2026-06-19 08:57:43.334 UTC [11186] LOG: shutting down
2026-06-19 08:57:43.491 UTC [11182] LOG: database system is shut down
2026-06-19T08:57:43Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (43993 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:57:43 dist-test-slave-h4g5 krb5kdc[12164](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:57:43 dist-test-slave-h4g5 krb5kdc[12164](info): set up 2 sockets
Jun 19 08:57:43 dist-test-slave-h4g5 krb5kdc[12164](info): commencing operation
krb5kdc: starting...
W20260619 08:57:44.523738 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 08:57:44 dist-test-slave-h4g5 krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:57:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:57:44Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260619 08:57:44.676067 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task5ocqKJ/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/postgres -l logfile start
2026-06-19 08:57:47.098 UTC [12194] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 08:57:47.098 UTC [12194] LOG: listening on IPv4 address "127.0.98.20", port 56325
2026-06-19 08:57:47.101 UTC [12194] LOG: listening on Unix socket "/tmp/.s.PGSQL.56325"
2026-06-19 08:57:47.128 UTC [12197] LOG: database system was shut down at 2026-06-19 08:57:45 UTC
2026-06-19 08:57:47.132 UTC [12194] LOG: database system is ready to accept connections
I20260619 08:57:47.991693 392 mini_postgres.cc:96] Postgres bound to 56325
2026-06-19 08:57:47.997 UTC [12205] FATAL: database "slave" does not exist
127.0.98.20:56325 - accepting connections
I20260619 08:57:47.997776 392 mini_ranger.cc:161] Starting Ranger...
I20260619 08:57:48.012184 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 08:57:48.490016 392 mini_ranger.cc:87] Created ranger Postgres database
I20260619 08:57:48.490087 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ranger-admin
2026-06-19 08:57:48,770 [I] DB FLAVOR :POSTGRES
2026-06-19 08:57:48,771 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 08:57:48,771 [I] Checking connection..
2026-06-19 08:57:48,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56325/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 08:57:49,019 [I] Checking connection passed.
2026-06-19 08:57:49,019 [I] --------- Verifying version history table ---------
2026-06-19 08:57:49,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56325/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 08:57:49,280 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 08:57:49,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56325/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 08:57:49,525 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 08:57:49,526 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 08:57:49,526 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56325/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 08:57:49.776 UTC [12269] WARNING: there is no transaction in progress
2026-06-19 08:57:49,782 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 08:57:49,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56325/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 08:57:50,014 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 08:57:50,014 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 08:57:50,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56325/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 08:57:50,246 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56325/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 08:57:50,504 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56325/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-h4g5.c.gcp-upstream.internal','N') ;"
2026-06-19 08:57:50,722 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56325/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 08:57:50,950 [I] Table x_portal_user does not exist in database ranger
2026-06-19 08:57:50,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56325/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 08:57:51,188 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 08:57:51,188 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56325/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 08:57:51,423 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56325/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 08:57:51,675 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 08:57:51,675 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56325/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 08:57:52.633 UTC [12377] WARNING: there is no transaction in progress
2026-06-19 08:57:52.649 UTC [12377] WARNING: there is no transaction in progress
2026-06-19 08:57:52.666 UTC [12377] WARNING: there is no transaction in progress
2026-06-19 08:57:52.681 UTC [12377] WARNING: there is no transaction in progress
2026-06-19 08:57:52.697 UTC [12377] WARNING: there is no transaction in progress
2026-06-19 08:57:52.714 UTC [12377] WARNING: there is no transaction in progress
2026-06-19 08:57:52.768 UTC [12377] WARNING: there is no transaction in progress
2026-06-19 08:57:52.775 UTC [12377] WARNING: there is no transaction in progress
2026-06-19 08:57:52.791 UTC [12377] WARNING: there is no transaction in progress
2026-06-19 08:57:52.801 UTC [12377] WARNING: there is no transaction in progress
2026-06-19 08:57:52.811 UTC [12377] WARNING: there is no transaction in progress
2026-06-19 08:57:52.825 UTC [12377] WARNING: there is no transaction in progress
2026-06-19 08:57:52.834 UTC [12377] WARNING: there is no transaction in progress
2026-06-19 08:57:52.840 UTC [12377] WARNING: there is no transaction in progress
2026-06-19 08:57:52.846 UTC [12377] WARNING: there is no transaction in progress
2026-06-19 08:57:52.852 UTC [12377] WARNING: there is no transaction in progress
2026-06-19 08:57:53,465 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 08:57:53,465 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56325/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 08:57:53,710 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 08:57:53,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56325/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 08:57:53,963 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 08:57:53,964 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56325/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 08:57:54,219 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 08:57:54,220 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56325/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 08:57:54,451 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56325/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 08:57:54,687 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56325/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-h4g5.c.gcp-upstream.internal';"
2026-06-19 08:57:54,909 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 08:57:54,909 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56325/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 08:57:55,133 [I] DB_PATCHES have already been applied
I20260619 08:57:55.138566 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task5ocqKJ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 08:57:55.138659 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260619 08:57:55.141595 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:37759
Jun 19, 2026 8:57:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 8:57:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 8:57:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 8:57:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 8:57:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 8:57:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 8:57:56 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 08:57:57.189702 392 mini_ranger.cc:160] Time spent starting Ranger: real 9.192s user 0.002s sys 0.008s
Jun 19, 2026 8:58:15 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 8:58:15 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 8:58:15 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 8:58:15 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 8:58:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 8:58:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 8:58:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 8:58:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 8:58:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 8:58:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 8:58:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 8:58:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 8:58:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 8:58:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 8:58:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 8:58:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 8:58:15 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 8:58:16 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 08:58:17.003397 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 08:58:17 dist-test-slave-h4g5 krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859497, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 08:58:17.085486 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 08:58:17.095196 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 08:58:18.000311 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 08:58:18.000378 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ranger-kms
I20260619 08:58:18.000394 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:56325
2026-06-19 08:58:18,145 [I] DB FLAVOR :POSTGRES
2026-06-19 08:58:18,145 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 08:58:18,145 [I] Checking connection
2026-06-19 08:58:18,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56325/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 08:58:18,396 [I] connection success
2026-06-19 08:58:18,396 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 08:58:18,396 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 08:58:18,396 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56325/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 08:58:18,660 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 08:58:18,660 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 08:58:18,660 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 08:58:18,661 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56325/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 08:58:18,988 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 08:58:19.424188 392 mini_ranger_kms.cc:310] Created kms service
I20260619 08:58:19.597237 392 mini_ranger_kms.cc:326] Created kudu user
I20260619 08:58:19.637121 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 08:58:19 dist-test-slave-h4g5 krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859497, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 08:58:20.043687 392 mini_ranger_kms.cc:384] Added ranger policy
I20260619 08:58:20.043797 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task5ocqKJ/build/release/bin/postgresql.jar:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task5ocqKJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 08:58:20.043814 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260619 08:58:20.046784 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12558
I20260619 08:58:20.046852 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:37563
Jun 19, 2026 8:58:20 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 8:58:20 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 8:58:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 8:58:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 8:58:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 8:58:21 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 08:58:22.058151 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.973s user 0.000s sys 0.127s
I20260619 08:58:22.058243 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 08:58:22.058290 392 mini_ranger_kms.cc:401] 127.0.98.20:37563/kms/v1/keys
Jun 19 08:58:22 dist-test-slave-h4g5 krb5kdc[12164](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781859502, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 08:58:22 dist-test-slave-h4g5 krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859502, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 08:58:22 dist-test-slave-h4g5 krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859502, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 08:58:25 dist-test-slave-h4g5 krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 08:58:25.114220 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39541
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39541
--ranger_config_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37563/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:58:25.192620 12609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:58:25.192826 12609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:58:25.192847 12609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:58:25.194034 12609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:58:25.194077 12609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:58:25.194090 12609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:58:25.194101 12609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:58:25.194111 12609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:58:25.195645 12609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:37563/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39541
--ranger_config_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:58:25.195861 12609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:58:25.196067 12609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:58:25.200776 12614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:25.200778 12615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:25.200769 12617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:25.201210 12609 server_base.cc:1034] running on GCE node
I20260619 08:58:25.201444 12609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:58:25.201705 12609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:58:25.202842 12609 hybrid_clock.cc:648] HybridClock initialized: now 1781859505202827 us; error 33 us; skew 500 ppm
Jun 19 08:58:25 dist-test-slave-h4g5 krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:58:25.205267 12609 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 08:58:25.205672 12609 webserver.cc:460] Webserver started at http://127.0.98.62:43405/ using document root <none> and password file <none>
I20260619 08:58:25.205873 12609 fs_manager.cc:359] Metadata directory not provided
I20260619 08:58:25.205914 12609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:58:25.206032 12609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 08:58:25 dist-test-slave-h4g5 krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 08:58:25.260371 12609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "0b43e79008f147caa0502f3615f5ea8b"
format_stamp: "Formatted at 2026-06-19 08:58:25 on dist-test-slave-h4g5"
server_key: "05b8e7943300b548a767c342719b2f3e"
server_key_iv: "1ee56ef8385b78baa1473797596702d6"
server_key_version: "kuduclusterkey@0"
I20260619 08:58:25.260753 12609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "0b43e79008f147caa0502f3615f5ea8b"
format_stamp: "Formatted at 2026-06-19 08:58:25 on dist-test-slave-h4g5"
server_key: "05b8e7943300b548a767c342719b2f3e"
server_key_iv: "1ee56ef8385b78baa1473797596702d6"
server_key_version: "kuduclusterkey@0"
I20260619 08:58:25.262107 12609 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:58:25.276158 12624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:25.276387 12609 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:58:25.276507 12609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "0b43e79008f147caa0502f3615f5ea8b"
format_stamp: "Formatted at 2026-06-19 08:58:25 on dist-test-slave-h4g5"
server_key: "05b8e7943300b548a767c342719b2f3e"
server_key_iv: "1ee56ef8385b78baa1473797596702d6"
server_key_version: "kuduclusterkey@0"
I20260619 08:58:25.276607 12609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:58:25.296047 12609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:58:25.298017 12609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:58:25.298156 12609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:58:25.302553 12609 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39541
I20260619 08:58:25.302623 12676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39541 every 8 connection(s)
I20260619 08:58:25.302961 12609 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:58:25.303689 12677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:58:25.306744 12677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b: Bootstrap starting.
I20260619 08:58:25.307440 12677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b: Neither blocks nor log segments found. Creating new log.
I20260619 08:58:25.307749 12677 log.cc:824] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b: Log is configured to *not* fsync() on all Append() calls
I20260619 08:58:25.308524 12677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b: No bootstrap required, opened a new log
I20260619 08:58:25.309753 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 12609
I20260619 08:58:25.309840 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 08:58:25.309995 392 external_mini_cluster.cc:1428] Setting key 2f92cdbe192a9f628d4de9685bb10514
I20260619 08:58:25.311491 12677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b43e79008f147caa0502f3615f5ea8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39541 } }
I20260619 08:58:25.311697 12677 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:58:25.311764 12677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b43e79008f147caa0502f3615f5ea8b, State: Initialized, Role: FOLLOWER
I20260619 08:58:25.311921 12677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b43e79008f147caa0502f3615f5ea8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39541 } }
I20260619 08:58:25.312004 12677 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:58:25.312034 12677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:58:25.312071 12677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:58:25.312805 12677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b43e79008f147caa0502f3615f5ea8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39541 } }
I20260619 08:58:25.312899 12677 leader_election.cc:302] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b43e79008f147caa0502f3615f5ea8b; no voters:
I20260619 08:58:25.313047 12677 leader_election.cc:288] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:58:25.313279 12677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:58:25.314673 12677 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 08:58:25.323114 12682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:58:25.326643 12682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b [term 1 LEADER]: Becoming Leader. State: Replica: 0b43e79008f147caa0502f3615f5ea8b, State: Running, Role: LEADER
I20260619 08:58:25.326874 12682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b43e79008f147caa0502f3615f5ea8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39541 } }
I20260619 08:58:25.327850 12683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b43e79008f147caa0502f3615f5ea8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b43e79008f147caa0502f3615f5ea8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39541 } } }
I20260619 08:58:25.328009 12683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b [sys.catalog]: This master's current role is: LEADER
I20260619 08:58:25.328274 12696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b43e79008f147caa0502f3615f5ea8b. Latest consensus state: current_term: 1 leader_uuid: "0b43e79008f147caa0502f3615f5ea8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b43e79008f147caa0502f3615f5ea8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39541 } } }
I20260619 08:58:25.328354 12696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b [sys.catalog]: This master's current role is: LEADER
Jun 19 08:58:25 dist-test-slave-h4g5 krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:58:25.330052 12699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:58:25.330631 12699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:58:25.332415 12699 catalog_manager.cc:1347] Generated new cluster ID: 56bd923b976d4888b9137757dfb0ad28
I20260619 08:58:25.332461 12699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:58:25.334368 12681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:25.310443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57704 (local address 127.0.98.62:39541)
0619 08:58:25.310612 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:25.310623 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:25.310656 (+ 33us) server_negotiation.cc:407] Connection header received
0619 08:58:25.311453 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:25.311459 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:25.311505 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:25.311625 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:25.312392 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:25.313823 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:25.314906 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:25.323575 (+ 8669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:25.323790 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:25.329922 (+ 6132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:25.329928 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:25.329929 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:25.329944 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:25.332056 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:25.332612 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:25.332616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:25.332616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:25.332719 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:25.332938 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:25.332939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:25.332939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:25.333272 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:25.333435 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:25.333979 (+ 544us) server_negotiation.cc:299] Negotiation successful
0619 08:58:25.334124 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
I20260619 08:58:25.342932 12699 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:58:25.343660 12699 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:58:25.355698 12699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b: Generated new TSK 0
I20260619 08:58:25.355916 12699 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 08:58:25.989082 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37563/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39541
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:58:26.096773 12722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:58:26.096976 12722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:58:26.097007 12722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:58:26.098317 12722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:58:26.098363 12722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:58:26.098397 12722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 08:58:26.099848 12722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:37563/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:58:26.100073 12722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:58:26.100311 12722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:58:26.105014 12730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:26.105014 12728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:26.105372 12727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:26.106210 12722 server_base.cc:1034] running on GCE node
I20260619 08:58:26.106457 12722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:58:26.106678 12722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:58:26.107852 12722 hybrid_clock.cc:648] HybridClock initialized: now 1781859506107848 us; error 38 us; skew 500 ppm
Jun 19 08:58:26 dist-test-slave-h4g5 krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:58:26.110723 12722 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 08:58:26.111182 12722 webserver.cc:460] Webserver started at http://127.0.98.1:34091/ using document root <none> and password file <none>
I20260619 08:58:26.111399 12722 fs_manager.cc:359] Metadata directory not provided
I20260619 08:58:26.111449 12722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:58:26.111575 12722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 08:58:26 dist-test-slave-h4g5 krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 08:58:26.123845 12722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-0/data/instance:
uuid: "e0e9a356ca7f43b1ad934ed8fe981240"
format_stamp: "Formatted at 2026-06-19 08:58:26 on dist-test-slave-h4g5"
server_key: "481371b07e8e7e4627dcd551e23340d5"
server_key_iv: "b50f5796a2ba5ac5ff7a22397b2df171"
server_key_version: "kuduclusterkey@0"
I20260619 08:58:26.124476 12722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e0e9a356ca7f43b1ad934ed8fe981240"
format_stamp: "Formatted at 2026-06-19 08:58:26 on dist-test-slave-h4g5"
server_key: "481371b07e8e7e4627dcd551e23340d5"
server_key_iv: "b50f5796a2ba5ac5ff7a22397b2df171"
server_key_version: "kuduclusterkey@0"
I20260619 08:58:26.126384 12722 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260619 08:58:26.145270 12738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:26.146059 12722 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:58:26.146226 12722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-0/wal
uuid: "e0e9a356ca7f43b1ad934ed8fe981240"
format_stamp: "Formatted at 2026-06-19 08:58:26 on dist-test-slave-h4g5"
server_key: "481371b07e8e7e4627dcd551e23340d5"
server_key_iv: "b50f5796a2ba5ac5ff7a22397b2df171"
server_key_version: "kuduclusterkey@0"
I20260619 08:58:26.146423 12722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:58:26.177256 12722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:58:26.179919 12722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:58:26.180104 12722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:58:26.180385 12722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:58:26.180867 12722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:58:26.180964 12722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:26.181005 12722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:58:26.181032 12722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:26.189141 12722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37509
I20260619 08:58:26.189234 12852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37509 every 8 connection(s)
I20260619 08:58:26.189810 12722 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
I20260619 08:58:26.196218 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 12722
Jun 19 08:58:26 dist-test-slave-h4g5 krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:58:26.196323 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-0/wal/instance
I20260619 08:58:26.196523 392 external_mini_cluster.cc:1428] Setting key 62395b9a54a4546c0df6ff7bc8196aff
I20260619 08:58:26.200032 12855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:26.190685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39541 (local address 127.0.98.1:43983)
0619 08:58:26.190897 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:26.190942 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:26.191460 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:26.191981 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:26.191985 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:26.192294 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:58:26.192520 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:26.192526 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:26.193268 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:26.193269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:26.194125 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:26.194128 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:26.194519 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:26.194520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:26.194639 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:26.195258 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:58:26.195267 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:58:26.196577 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:58:26.198475 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:26.198482 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:26.198485 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:26.198795 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:26.199032 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:26.199033 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:26.199034 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:26.199162 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:26.199440 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:58:26.199445 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:58:26.199616 (+ 171us) client_negotiation.cc:769] Sending connection context
0619 08:58:26.199720 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 08:58:26.199865 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":83,"threads_started":1}
I20260619 08:58:26.200459 12856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:26.190708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43983 (local address 127.0.98.62:39541)
0619 08:58:26.191673 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:26.191675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:26.191685 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:58:26.191714 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:26.191715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:26.191729 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:26.191830 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:26.192633 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:26.193172 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:26.194219 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:26.194429 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:26.194492 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:26.196675 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:26.196676 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:26.196676 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:26.196684 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:26.198379 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:26.198885 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:26.198886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:26.198887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:26.198967 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:26.199223 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:26.199224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:26.199225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:26.199358 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:26.199416 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:26.200286 (+ 870us) server_negotiation.cc:299] Negotiation successful
0619 08:58:26.200373 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":923,"thread_start_us":111,"threads_started":1}
I20260619 08:58:26.200901 12853 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39541
I20260619 08:58:26.200997 12853 heartbeater.cc:464] Registering TS with master...
I20260619 08:58:26.201185 12853 heartbeater.cc:511] Master 127.0.98.62:39541 requested a full tablet report, sending...
I20260619 08:58:26.201691 12641 ts_manager.cc:194] Registered new tserver with Master: e0e9a356ca7f43b1ad934ed8fe981240 (127.0.98.1:37509)
I20260619 08:58:26.202528 12641 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43983
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 08:58:26.222280 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37563/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39541
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 08:58:26 dist-test-slave-h4g5 krb5kdc[12164](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781859506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 08:58:26.342465 12859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:58:26.342772 12859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:58:26.342867 12859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:58:26.344779 12859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:58:26.344848 12859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:58:26.344928 12859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 08:58:26.347226 12859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:37563/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:58:26.347692 12859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:58:26.348014 12859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:58:26.354527 12866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:26.354840 12867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:26.357100 12869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:26.358914 12859 server_base.cc:1034] running on GCE node
I20260619 08:58:26.359130 12859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:58:26.359385 12859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:58:26.360531 12859 hybrid_clock.cc:648] HybridClock initialized: now 1781859506360520 us; error 25 us; skew 500 ppm
Jun 19 08:58:26 dist-test-slave-h4g5 krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:58:26.363838 12859 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 08:58:26.364347 12859 webserver.cc:460] Webserver started at http://127.0.98.2:38043/ using document root <none> and password file <none>
I20260619 08:58:26.364560 12859 fs_manager.cc:359] Metadata directory not provided
I20260619 08:58:26.364606 12859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:58:26.364748 12859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 08:58:26 dist-test-slave-h4g5 krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 08:58:26.377998 12859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-1/data/instance:
uuid: "1355b420aa974adaa4762429a384db6c"
format_stamp: "Formatted at 2026-06-19 08:58:26 on dist-test-slave-h4g5"
server_key: "839c7c3343798ccb89412a9ee36f0b53"
server_key_iv: "1c6770f346bf529ecc61713dcf4c8c21"
server_key_version: "kuduclusterkey@0"
I20260619 08:58:26.378611 12859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1355b420aa974adaa4762429a384db6c"
format_stamp: "Formatted at 2026-06-19 08:58:26 on dist-test-slave-h4g5"
server_key: "839c7c3343798ccb89412a9ee36f0b53"
server_key_iv: "1c6770f346bf529ecc61713dcf4c8c21"
server_key_version: "kuduclusterkey@0"
I20260619 08:58:26.381191 12859 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260619 08:58:26.400238 12877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:26.400406 12859 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:58:26.400496 12859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-1/wal
uuid: "1355b420aa974adaa4762429a384db6c"
format_stamp: "Formatted at 2026-06-19 08:58:26 on dist-test-slave-h4g5"
server_key: "839c7c3343798ccb89412a9ee36f0b53"
server_key_iv: "1c6770f346bf529ecc61713dcf4c8c21"
server_key_version: "kuduclusterkey@0"
I20260619 08:58:26.400588 12859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:58:26.421873 12859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:58:26.424216 12859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:58:26.424366 12859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:58:26.424592 12859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:58:26.424947 12859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:58:26.424989 12859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:26.425022 12859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:58:26.425045 12859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:26.430981 12859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39439
I20260619 08:58:26.431388 12859 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
I20260619 08:58:26.434491 12990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39439 every 8 connection(s)
I20260619 08:58:26.438742 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 12859
I20260619 08:58:26.438831 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-1/wal/instance
I20260619 08:58:26.439013 392 external_mini_cluster.cc:1428] Setting key a9b656196953a6e1a36b00b4c9452179
Jun 19 08:58:26 dist-test-slave-h4g5 krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:58:26.447387 12993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:26.436589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39541 (local address 127.0.98.2:44305)
0619 08:58:26.436742 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:26.436785 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:26.437277 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:26.437517 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:26.437522 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:26.437701 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:58:26.437893 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:26.437897 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:26.438558 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:26.438559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:26.441067 (+ 2508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:26.441071 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:26.441617 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:26.441618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:26.441767 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:26.442362 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:58:26.442374 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:58:26.443954 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:58:26.445755 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:26.445762 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:26.445764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:26.446062 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:26.446270 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:26.446271 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:26.446271 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:26.446400 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:26.446688 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:58:26.446694 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:58:26.446870 (+ 176us) client_negotiation.cc:769] Sending connection context
0619 08:58:26.446992 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 08:58:26.447148 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 08:58:26.448189 12856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:26.436601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44305 (local address 127.0.98.62:39541)
0619 08:58:26.436653 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:26.436655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:26.436888 (+ 233us) server_negotiation.cc:407] Connection header received
0619 08:58:26.437372 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:26.437374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:26.437386 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:26.437432 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:26.437984 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:26.438476 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:26.441183 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:26.441476 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:26.441535 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:26.444061 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:26.444062 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:26.444063 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:26.444071 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:26.445665 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:26.446147 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:26.446148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:26.446149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:26.446206 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:26.446469 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:26.446470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:26.446470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:26.446606 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:26.446663 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:26.447989 (+ 1326us) server_negotiation.cc:299] Negotiation successful
0619 08:58:26.448077 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 08:58:26.448796 12991 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39541
I20260619 08:58:26.448894 12991 heartbeater.cc:464] Registering TS with master...
I20260619 08:58:26.449070 12991 heartbeater.cc:511] Master 127.0.98.62:39541 requested a full tablet report, sending...
I20260619 08:58:26.449438 12641 ts_manager.cc:194] Registered new tserver with Master: 1355b420aa974adaa4762429a384db6c (127.0.98.2:39439)
I20260619 08:58:26.449800 12641 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44305
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 08:58:26.486753 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37563/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39541
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:58:26.614369 12996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:58:26.614614 12996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:58:26.614650 12996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:58:26.615835 12996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:58:26.615880 12996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:58:26.615914 12996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 08:58:26.617455 12996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:37563/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:58:26.617683 12996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:58:26.617903 12996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:58:26.622617 13004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:26.622685 13002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:26.622625 13001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:26.623332 12996 server_base.cc:1034] running on GCE node
I20260619 08:58:26.623534 12996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:58:26.623760 12996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:58:26.624907 12996 hybrid_clock.cc:648] HybridClock initialized: now 1781859506624891 us; error 28 us; skew 500 ppm
Jun 19 08:58:26 dist-test-slave-h4g5 krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:58:26.627298 12996 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 08:58:26.627749 12996 webserver.cc:460] Webserver started at http://127.0.98.3:35361/ using document root <none> and password file <none>
I20260619 08:58:26.627964 12996 fs_manager.cc:359] Metadata directory not provided
I20260619 08:58:26.628010 12996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:58:26.628209 12996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 08:58:26 dist-test-slave-h4g5 krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 08:58:26.641726 12996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-2/data/instance:
uuid: "8bfb1ede8b9d4700ba75b36ff37f315e"
format_stamp: "Formatted at 2026-06-19 08:58:26 on dist-test-slave-h4g5"
server_key: "2da61171cbc1e39f01436294d471eaa2"
server_key_iv: "9623207a13177405445cb5d7ee48eeee"
server_key_version: "kuduclusterkey@0"
I20260619 08:58:26.642266 12996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8bfb1ede8b9d4700ba75b36ff37f315e"
format_stamp: "Formatted at 2026-06-19 08:58:26 on dist-test-slave-h4g5"
server_key: "2da61171cbc1e39f01436294d471eaa2"
server_key_iv: "9623207a13177405445cb5d7ee48eeee"
server_key_version: "kuduclusterkey@0"
I20260619 08:58:26.655748 12996 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.001s sys 0.000s
I20260619 08:58:26.678540 13011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:26.678781 12996 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:58:26.678862 12996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-2/wal
uuid: "8bfb1ede8b9d4700ba75b36ff37f315e"
format_stamp: "Formatted at 2026-06-19 08:58:26 on dist-test-slave-h4g5"
server_key: "2da61171cbc1e39f01436294d471eaa2"
server_key_iv: "9623207a13177405445cb5d7ee48eeee"
server_key_version: "kuduclusterkey@0"
I20260619 08:58:26.678948 12996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:58:26.713667 12996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:58:26.716005 12996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:58:26.716235 12996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:58:26.716481 12996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:58:26.716871 12996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:58:26.716950 12996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:26.717027 12996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:58:26.717090 12996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:26.723141 12996 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35753
I20260619 08:58:26.723618 12996 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
I20260619 08:58:26.733131 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 12996
I20260619 08:58:26.733229 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-2/wal/instance
I20260619 08:58:26.733403 392 external_mini_cluster.cc:1428] Setting key 078c3b5be1ebc9b52b6948befe5bc088
I20260619 08:58:26.736515 13124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35753 every 8 connection(s)
Jun 19 08:58:26 dist-test-slave-h4g5 krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:58:26.748474 12856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:26.737264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39989 (local address 127.0.98.62:39541)
0619 08:58:26.737309 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:26.737311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:26.737627 (+ 316us) server_negotiation.cc:407] Connection header received
0619 08:58:26.738280 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:26.738281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:26.738296 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:26.738387 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:26.739260 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:26.740135 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:26.741594 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:26.741876 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:26.741915 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:26.745240 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:26.745241 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:26.745242 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:26.745249 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:26.746817 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:26.747341 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:26.747342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:26.747343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:26.747416 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:26.747716 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:26.747716 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:26.747717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:26.747847 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:26.747884 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:26.748301 (+ 417us) server_negotiation.cc:299] Negotiation successful
0619 08:58:26.748382 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 08:58:26.749229 13127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:26.737280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39541 (local address 127.0.98.3:39989)
0619 08:58:26.737491 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:26.737543 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:26.738159 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:26.738463 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:26.738468 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:26.738787 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:58:26.739135 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:26.739144 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:26.740259 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:26.740260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:26.741493 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:26.741496 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:26.742033 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:26.742034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:26.742181 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:26.742725 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:58:26.742774 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:58:26.745141 (+ 2367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:58:26.746922 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:26.746931 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:26.746933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:26.747247 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:26.747492 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:26.747493 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:26.747493 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:26.747643 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:26.747925 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:58:26.747931 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:58:26.748153 (+ 222us) client_negotiation.cc:769] Sending connection context
0619 08:58:26.748818 (+ 665us) client_negotiation.cc:240] Negotiation successful
0619 08:58:26.749000 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":101,"threads_started":1}
I20260619 08:58:26.749852 13125 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39541
I20260619 08:58:26.749944 13125 heartbeater.cc:464] Registering TS with master...
I20260619 08:58:26.750135 13125 heartbeater.cc:511] Master 127.0.98.62:39541 requested a full tablet report, sending...
I20260619 08:58:26.750540 12641 ts_manager.cc:194] Registered new tserver with Master: 8bfb1ede8b9d4700ba75b36ff37f315e (127.0.98.3:35753)
I20260619 08:58:26.751020 12641 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39989
I20260619 08:58:26.756623 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:58:26.757329 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 12722
I20260619 08:58:26.762641 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 12859
I20260619 08:58:26.768411 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 12996
I20260619 08:58:26.772259 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 12609
I20260619 08:58:26.778663 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39541
--webserver_interface=127.0.98.62
--webserver_port=43405
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39541
--ranger_config_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37563/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37563/kms with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:58:26.858124 13128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:58:26.858364 13128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:58:26.858386 13128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:58:26.859587 13128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:58:26.859640 13128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:58:26.859653 13128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:58:26.859664 13128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:58:26.859674 13128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:58:26.861413 13128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37563/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39541
--ranger_config_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:58:26.861676 13128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:58:26.861922 13128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:58:26.867048 13134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:26.867177 13128 server_base.cc:1034] running on GCE node
W20260619 08:58:26.867057 13136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:26.867048 13133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:26.867484 13128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:58:26.867789 13128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:58:26.868952 13128 hybrid_clock.cc:648] HybridClock initialized: now 1781859506868922 us; error 45 us; skew 500 ppm
Jun 19 08:58:26 dist-test-slave-h4g5 krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:58:26.871570 13128 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 08:58:26.872135 13128 webserver.cc:460] Webserver started at http://127.0.98.62:43405/ using document root <none> and password file <none>
I20260619 08:58:26.872370 13128 fs_manager.cc:359] Metadata directory not provided
I20260619 08:58:26.872435 13128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 08:58:26 dist-test-slave-h4g5 krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 08:58:26.949756 13128 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:58:26.950536 13144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:26.950706 13128 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:58:26.950783 13128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "0b43e79008f147caa0502f3615f5ea8b"
format_stamp: "Formatted at 2026-06-19 08:58:25 on dist-test-slave-h4g5"
server_key: "05b8e7943300b548a767c342719b2f3e"
server_key_iv: "1ee56ef8385b78baa1473797596702d6"
server_key_version: "kuduclusterkey@0"
I20260619 08:58:26.951143 13128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:58:26.970009 13128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:58:26.972059 13128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:58:26.972252 13128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:58:26.976740 13128 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39541
I20260619 08:58:26.976823 13196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39541 every 8 connection(s)
I20260619 08:58:26.977171 13128 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:58:26.978819 13197 sys_catalog.cc:263] Verifying existing consensus state
I20260619 08:58:26.979691 13197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b: Bootstrap starting.
I20260619 08:58:26.984862 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 13128
I20260619 08:58:26.985152 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37563/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37509
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39541
--webserver_port=34091
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37563/kms with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 08:58:26.989581 13197 log.cc:824] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b: Log is configured to *not* fsync() on all Append() calls
I20260619 08:58:26.991451 13197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 08:58:26.991729 13197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b: Bootstrap complete.
I20260619 08:58:26.993969 13197 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b43e79008f147caa0502f3615f5ea8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39541 } }
I20260619 08:58:26.994202 13197 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b43e79008f147caa0502f3615f5ea8b, State: Initialized, Role: FOLLOWER
I20260619 08:58:26.994357 13197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b43e79008f147caa0502f3615f5ea8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39541 } }
I20260619 08:58:26.994428 13197 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:58:26.994474 13197 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:58:26.994529 13197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b [term 1 FOLLOWER]: Advancing to term 2
I20260619 08:58:26.995543 13197 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b43e79008f147caa0502f3615f5ea8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39541 } }
I20260619 08:58:26.995638 13197 leader_election.cc:302] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b43e79008f147caa0502f3615f5ea8b; no voters:
I20260619 08:58:26.995813 13197 leader_election.cc:288] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 08:58:26.995880 13201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b [term 2 FOLLOWER]: Leader election won for term 2
I20260619 08:58:26.995987 13201 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b [term 2 LEADER]: Becoming Leader. State: Replica: 0b43e79008f147caa0502f3615f5ea8b, State: Running, Role: LEADER
I20260619 08:58:26.996099 13201 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b43e79008f147caa0502f3615f5ea8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39541 } }
I20260619 08:58:26.996099 13197 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:58:26.996423 13202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0b43e79008f147caa0502f3615f5ea8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b43e79008f147caa0502f3615f5ea8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39541 } } }
I20260619 08:58:26.996431 13203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b43e79008f147caa0502f3615f5ea8b. Latest consensus state: current_term: 2 leader_uuid: "0b43e79008f147caa0502f3615f5ea8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b43e79008f147caa0502f3615f5ea8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39541 } } }
I20260619 08:58:26.996539 13202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b [sys.catalog]: This master's current role is: LEADER
I20260619 08:58:26.996542 13203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b [sys.catalog]: This master's current role is: LEADER
I20260619 08:58:26.996729 13205 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:58:26.997195 13205 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:58:26.997311 13205 catalog_manager.cc:1259] Loaded cluster ID: 56bd923b976d4888b9137757dfb0ad28
I20260619 08:58:26.997342 13205 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:58:26.997779 13197 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 08:58:27.000174 13205 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:58:27.000768 13205 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b: Loaded TSK: 0
I20260619 08:58:27.001097 13205 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 08:58:27.070492 13199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:58:27.070859 13199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:58:27.070964 13199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:58:27.072808 13199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:58:27.072921 13199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:58:27.073019 13199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 08:58:27.075186 13199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37563/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37509
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34091
--tserver_master_addrs=127.0.98.62:39541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:58:27.075438 13199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:58:27.075698 13199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:58:27.082067 13234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:27.082091 13235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:27.082304 13199 server_base.cc:1034] running on GCE node
W20260619 08:58:27.083585 13237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:27.083832 13199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:58:27.084151 13199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:58:27.085330 13199 hybrid_clock.cc:648] HybridClock initialized: now 1781859507085298 us; error 43 us; skew 500 ppm
Jun 19 08:58:27 dist-test-slave-h4g5 krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:58:27.088619 13199 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 08:58:27.089186 13199 webserver.cc:460] Webserver started at http://127.0.98.1:34091/ using document root <none> and password file <none>
I20260619 08:58:27.089484 13199 fs_manager.cc:359] Metadata directory not provided
I20260619 08:58:27.089570 13199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 08:58:27 dist-test-slave-h4g5 krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 08:58:27.117923 13199 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260619 08:58:27.118850 13245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:27.119016 13199 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:58:27.119108 13199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-0/wal
uuid: "e0e9a356ca7f43b1ad934ed8fe981240"
format_stamp: "Formatted at 2026-06-19 08:58:26 on dist-test-slave-h4g5"
server_key: "481371b07e8e7e4627dcd551e23340d5"
server_key_iv: "b50f5796a2ba5ac5ff7a22397b2df171"
server_key_version: "kuduclusterkey@0"
I20260619 08:58:27.119581 13199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:58:27.150230 13199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:58:27.152719 13199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:58:27.152918 13199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:58:27.153198 13199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:58:27.153605 13199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:58:27.153656 13199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:27.153687 13199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:58:27.153703 13199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:27.162355 13199 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37509
I20260619 08:58:27.162534 13358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37509 every 8 connection(s)
I20260619 08:58:27.162851 13199 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
I20260619 08:58:27.164561 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 13199
I20260619 08:58:27.164825 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37563/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39439
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39541
--webserver_port=38043
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37563/kms with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 08:58:27 dist-test-slave-h4g5 krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:58:27.178483 13361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:27.163604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39541 (local address 127.0.98.1:60037)
0619 08:58:27.163867 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:27.163932 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:27.166956 (+ 3024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:27.167567 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:27.167572 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:27.167804 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:58:27.168063 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:27.168069 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:27.169754 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:27.169755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:27.171123 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:27.171127 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:27.171976 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:27.171977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:27.172169 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:27.172794 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:58:27.172804 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:58:27.174474 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:58:27.176679 (+ 2205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:27.176686 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:27.176688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:27.176976 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:27.177234 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:27.177235 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:27.177235 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:27.177374 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:27.177822 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:58:27.177828 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:58:27.177988 (+ 160us) client_negotiation.cc:769] Sending connection context
0619 08:58:27.178077 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 08:58:27.178237 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":82,"threads_started":1}
I20260619 08:58:27.179505 13362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:27.163672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60037 (local address 127.0.98.62:39541)
0619 08:58:27.164266 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:27.164277 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:27.164327 (+ 50us) server_negotiation.cc:407] Connection header received
0619 08:58:27.167279 (+ 2952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:27.167286 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:27.167347 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:27.167505 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:27.168197 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:27.169647 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:27.171226 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:27.171862 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:27.171975 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:27.174607 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:27.174612 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:27.174612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:27.174625 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:27.176589 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:27.177063 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:27.177066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:27.177067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:27.177172 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:27.177436 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:27.177437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:27.177437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:27.177728 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:27.177796 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:27.179101 (+ 1305us) server_negotiation.cc:299] Negotiation successful
0619 08:58:27.179291 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":99,"threads_started":1}
I20260619 08:58:27.180493 13359 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39541
I20260619 08:58:27.180584 13359 heartbeater.cc:464] Registering TS with master...
I20260619 08:58:27.180760 13359 heartbeater.cc:511] Master 127.0.98.62:39541 requested a full tablet report, sending...
W20260619 08:58:27.181605 13359 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39541 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 08:58:27.181668 13359 heartbeater.cc:464] Registering TS with master...
I20260619 08:58:27.181722 13359 heartbeater.cc:511] Master 127.0.98.62:39541 requested a full tablet report, sending...
I20260619 08:58:27.182072 13359 heartbeater.cc:464] Registering TS with master...
I20260619 08:58:27.182138 13359 heartbeater.cc:511] Master 127.0.98.62:39541 requested a full tablet report, sending...
W20260619 08:58:27.182446 13359 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 08:58:27.271834 13363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:58:27.272151 13363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:58:27.272192 13363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:58:27.273885 13363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:58:27.273948 13363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:58:27.273990 13363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 08:58:27.275760 13363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37563/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39439
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38043
--tserver_master_addrs=127.0.98.62:39541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:58:27.276026 13363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:58:27.276338 13363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:58:27.282574 13371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:27.282545 13368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:27.282546 13369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:27.284060 13363 server_base.cc:1034] running on GCE node
I20260619 08:58:27.284361 13363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:58:27.284734 13363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:58:27.285976 13363 hybrid_clock.cc:648] HybridClock initialized: now 1781859507285953 us; error 34 us; skew 500 ppm
Jun 19 08:58:27 dist-test-slave-h4g5 krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:58:27.288653 13363 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 08:58:27.289193 13363 webserver.cc:460] Webserver started at http://127.0.98.2:38043/ using document root <none> and password file <none>
I20260619 08:58:27.289474 13363 fs_manager.cc:359] Metadata directory not provided
I20260619 08:58:27.289546 13363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 08:58:27 dist-test-slave-h4g5 krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 08:58:27.312240 13363 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260619 08:58:27.313313 13379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:27.313539 13363 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:58:27.313649 13363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-1/wal
uuid: "1355b420aa974adaa4762429a384db6c"
format_stamp: "Formatted at 2026-06-19 08:58:26 on dist-test-slave-h4g5"
server_key: "839c7c3343798ccb89412a9ee36f0b53"
server_key_iv: "1c6770f346bf529ecc61713dcf4c8c21"
server_key_version: "kuduclusterkey@0"
I20260619 08:58:27.314190 13363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:58:27.334259 13363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:58:27.336884 13363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:58:27.337049 13363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:58:27.337311 13363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:58:27.337689 13363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:58:27.337741 13363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:27.337778 13363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:58:27.337801 13363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:27.345768 13363 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39439
I20260619 08:58:27.345956 13492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39439 every 8 connection(s)
I20260619 08:58:27.346359 13363 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
I20260619 08:58:27.352151 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 13363
I20260619 08:58:27.352559 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37563/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35753
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39541
--webserver_port=35361
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37563/kms with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 08:58:27 dist-test-slave-h4g5 krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:58:27.357916 13362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:27.347274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41607 (local address 127.0.98.62:39541)
0619 08:58:27.347333 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:27.347336 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:27.347589 (+ 253us) server_negotiation.cc:407] Connection header received
0619 08:58:27.348163 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:27.348165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:27.348179 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:27.348279 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:27.349127 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:27.349929 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:27.351031 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:27.351348 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:27.351447 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:27.354352 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:27.354355 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:27.354355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:27.354363 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:27.356122 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:27.356685 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:27.356686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:27.356686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:27.356755 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:27.357096 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:27.357097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:27.357098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:27.357267 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:27.357348 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:27.357725 (+ 377us) server_negotiation.cc:299] Negotiation successful
0619 08:58:27.357820 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 08:58:27.358125 13495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:27.347255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39541 (local address 127.0.98.2:41607)
0619 08:58:27.347435 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:27.347480 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:27.347983 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:27.348364 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:27.348369 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:27.348693 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:58:27.349003 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:27.349011 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:27.350048 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:27.350048 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:27.350932 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:27.350934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:27.351465 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:27.351465 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:27.351608 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:27.352473 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:58:27.352485 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:58:27.354215 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:58:27.356204 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:27.356213 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:27.356216 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:27.356597 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:27.356837 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:27.356838 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:27.356838 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:27.357004 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:27.357347 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:58:27.357353 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:58:27.357560 (+ 207us) client_negotiation.cc:769] Sending connection context
0619 08:58:27.357681 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 08:58:27.357859 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":84,"threads_started":1}
I20260619 08:58:27.358704 13493 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39541
I20260619 08:58:27.358807 13493 heartbeater.cc:464] Registering TS with master...
I20260619 08:58:27.359018 13493 heartbeater.cc:511] Master 127.0.98.62:39541 requested a full tablet report, sending...
W20260619 08:58:27.359773 13493 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39541 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 08:58:27.359849 13493 heartbeater.cc:464] Registering TS with master...
I20260619 08:58:27.359917 13493 heartbeater.cc:511] Master 127.0.98.62:39541 requested a full tablet report, sending...
I20260619 08:58:27.360332 13493 heartbeater.cc:464] Registering TS with master...
I20260619 08:58:27.360428 13493 heartbeater.cc:511] Master 127.0.98.62:39541 requested a full tablet report, sending...
W20260619 08:58:27.360783 13493 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 08:58:27.460129 13496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:58:27.460398 13496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:58:27.460434 13496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:58:27.462141 13496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:58:27.462205 13496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:58:27.462248 13496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 08:58:27.464444 13496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37563/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35753
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35361
--tserver_master_addrs=127.0.98.62:39541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:58:27.464715 13496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:58:27.464990 13496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:58:27.471225 13501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:27.471336 13504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:27.471232 13502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:27.471781 13496 server_base.cc:1034] running on GCE node
I20260619 08:58:27.471997 13496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:58:27.472286 13496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:58:27.473445 13496 hybrid_clock.cc:648] HybridClock initialized: now 1781859507473427 us; error 32 us; skew 500 ppm
Jun 19 08:58:27 dist-test-slave-h4g5 krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:58:27.476553 13496 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 08:58:27.477102 13496 webserver.cc:460] Webserver started at http://127.0.98.3:35361/ using document root <none> and password file <none>
I20260619 08:58:27.477346 13496 fs_manager.cc:359] Metadata directory not provided
I20260619 08:58:27.477421 13496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 08:58:27 dist-test-slave-h4g5 krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 08:58:27.496958 13496 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:58:27.498023 13513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:27.498251 13496 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:58:27.498355 13496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-2/wal
uuid: "8bfb1ede8b9d4700ba75b36ff37f315e"
format_stamp: "Formatted at 2026-06-19 08:58:26 on dist-test-slave-h4g5"
server_key: "2da61171cbc1e39f01436294d471eaa2"
server_key_iv: "9623207a13177405445cb5d7ee48eeee"
server_key_version: "kuduclusterkey@0"
I20260619 08:58:27.498697 13496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:58:27.518980 13496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:58:27.521111 13496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:58:27.521232 13496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:58:27.521492 13496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:58:27.521871 13496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:58:27.521925 13496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:27.521957 13496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:58:27.521975 13496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:27.528499 13496 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35753
I20260619 08:58:27.528975 13496 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
I20260619 08:58:27.529496 13626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35753 every 8 connection(s)
I20260619 08:58:27.530187 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 13496
Jun 19 08:58:27 dist-test-slave-h4g5 krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:58:27.538206 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:27.530559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57716 (local address 127.0.98.62:39541)
0619 08:58:27.530854 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:27.530856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:27.530862 (+ 6us) server_negotiation.cc:407] Connection header received
0619 08:58:27.530879 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:27.530880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:27.530888 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:27.530944 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:27.531683 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:27.532233 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:27.533593 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:27.533855 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:27.533925 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:27.535285 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:27.535286 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:27.535287 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:27.535294 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:27.537019 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:27.537442 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:27.537443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:27.537444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:27.537501 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:27.537709 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:27.537709 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:27.537710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:27.537837 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:27.537989 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:27.538026 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 08:58:27.538105 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":55,"threads_started":1}
I20260619 08:58:27.539713 13362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:27.529706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32981 (local address 127.0.98.62:39541)
0619 08:58:27.529756 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:27.529758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:27.529966 (+ 208us) server_negotiation.cc:407] Connection header received
0619 08:58:27.530564 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:27.530566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:27.530578 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:27.530610 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:27.531448 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:27.532249 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:27.533704 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:27.533951 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:27.533999 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:27.536306 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:27.536307 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:27.536307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:27.536314 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:27.537788 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:27.538273 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:27.538274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:27.538274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:27.538342 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:27.538996 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:27.538996 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:27.538997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:27.539136 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:27.539189 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:27.539586 (+ 397us) server_negotiation.cc:299] Negotiation successful
0619 08:58:27.539651 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 08:58:27.539929 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:27.529667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39541 (local address 127.0.98.3:32981)
0619 08:58:27.529834 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:27.529876 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:27.530432 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:27.530806 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:27.530811 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:27.531053 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:58:27.531345 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:27.531352 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:27.532362 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:27.532363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:27.533619 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:27.533623 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:27.534037 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:27.534038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:27.534151 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:27.534606 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:58:27.534615 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:58:27.536221 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:58:27.537857 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:27.537865 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:27.537867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:27.538177 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:27.538519 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:27.538520 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:27.538520 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:27.538764 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:27.539207 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:58:27.539213 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:58:27.539399 (+ 186us) client_negotiation.cc:769] Sending connection context
0619 08:58:27.539534 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 08:58:27.539694 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
I20260619 08:58:27.540491 13627 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39541
I20260619 08:58:27.540597 13627 heartbeater.cc:464] Registering TS with master...
I20260619 08:58:27.540797 13627 heartbeater.cc:511] Master 127.0.98.62:39541 requested a full tablet report, sending...
W20260619 08:58:27.544803 13627 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39541 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 08:58:27.544891 13627 heartbeater.cc:464] Registering TS with master...
I20260619 08:58:27.544967 13627 heartbeater.cc:511] Master 127.0.98.62:39541 requested a full tablet report, sending...
I20260619 08:58:27.545396 13627 heartbeater.cc:464] Registering TS with master...
I20260619 08:58:27.545491 13627 heartbeater.cc:511] Master 127.0.98.62:39541 requested a full tablet report, sending...
W20260619 08:58:27.545830 13627 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 08:58:28 dist-test-slave-h4g5 krb5kdc[12164](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781859508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:58:28.182637 13359 heartbeater.cc:464] Registering TS with master...
I20260619 08:58:28.182794 13359 heartbeater.cc:511] Master 127.0.98.62:39541 requested a full tablet report, sending...
I20260619 08:58:28.183161 13161 ts_manager.cc:194] Registered new tserver with Master: e0e9a356ca7f43b1ad934ed8fe981240 (127.0.98.1:37509)
I20260619 08:58:28.183828 13161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60037
I20260619 08:58:28.360980 13493 heartbeater.cc:464] Registering TS with master...
I20260619 08:58:28.361155 13493 heartbeater.cc:511] Master 127.0.98.62:39541 requested a full tablet report, sending...
I20260619 08:58:28.361507 13161 ts_manager.cc:194] Registered new tserver with Master: 1355b420aa974adaa4762429a384db6c (127.0.98.2:39439)
I20260619 08:58:28.362074 13161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41607
Jun 19 08:58:28 dist-test-slave-h4g5 krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 08:58:28.546036 13627 heartbeater.cc:464] Registering TS with master...
I20260619 08:58:28.546200 13627 heartbeater.cc:511] Master 127.0.98.62:39541 requested a full tablet report, sending...
I20260619 08:58:28.546563 13161 ts_manager.cc:194] Registered new tserver with Master: 8bfb1ede8b9d4700ba75b36ff37f315e (127.0.98.3:35753)
I20260619 08:58:28.547096 13161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:32981
I20260619 08:58:28.552287 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:58:28.563764 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:28.553242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57718 (local address 127.0.98.62:39541)
0619 08:58:28.553373 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:28.553375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:28.556325 (+ 2950us) server_negotiation.cc:407] Connection header received
0619 08:58:28.556389 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:28.556390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:28.556403 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:28.556483 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:28.557203 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:28.557998 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:28.559007 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:28.559337 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:28.559402 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:28.560905 (+ 1503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:28.560907 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:28.560907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:28.560916 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:28.562592 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:28.562973 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:28.562973 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:28.562974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:28.563031 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:28.563219 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:28.563220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:28.563220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:28.563351 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:28.563444 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:28.563578 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 08:58:28.563657 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":78,"threads_started":1}
I20260619 08:58:28.565956 13161 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:58:28.568234 13161 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:58:28.577087 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:28.571381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37509 (local address 127.0.0.1:37390)
0619 08:58:28.571748 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:28.572038 (+ 290us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:28.572133 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:28.572622 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:28.572625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:28.572627 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:58:28.572706 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:28.572709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:28.574207 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:28.574208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:28.575506 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:28.575508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:28.576778 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:28.576779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:28.576860 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:28.576873 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 08:58:28.576937 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 08:58:28.576997 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":3,"thread_start_us":76,"threads_started":1}
I20260619 08:58:28.577325 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:28.572129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37390 (local address 127.0.98.1:37509)
0619 08:58:28.572479 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:28.572483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:28.572492 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:58:28.572520 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:28.572521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:28.572534 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:28.572605 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:58:28.572788 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:28.574133 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:28.575597 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:28.576620 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:28.576679 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:28.576745 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:28.577209 (+ 464us) server_negotiation.cc:299] Negotiation successful
0619 08:58:28.577251 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":46,"threads_started":1}
I20260619 08:58:28.578076 13275 tablet_service.cc:1467] Processing CreateTablet for tablet d0e8f8e39999403081bdae5cf5acff22 (DEFAULT_TABLE table=test-table [id=3dd9ea79795b44fc8ee26c9d4d2e9f52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:58:28.578436 13275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0e8f8e39999403081bdae5cf5acff22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:58:28.580456 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:28.571381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35753 (local address 127.0.0.1:42786)
0619 08:58:28.572703 (+ 1322us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:28.572712 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:28.572754 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:28.573285 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:28.573286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:28.573288 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:58:28.573324 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:28.573326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:28.574551 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:28.574551 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:28.575582 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:28.575584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:28.580160 (+ 4576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:28.580162 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:28.580234 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:28.580240 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 08:58:28.580312 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 08:58:28.580363 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1304,"mutex_wait_us":1,"thread_start_us":77,"threads_started":1}
I20260619 08:58:28.580844 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:28.571953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42786 (local address 127.0.98.3:35753)
0619 08:58:28.572411 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:28.572414 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:28.572807 (+ 393us) server_negotiation.cc:407] Connection header received
0619 08:58:28.572840 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:28.572843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:28.572857 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:28.572917 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:58:28.573389 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:28.574356 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:28.575656 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:28.576361 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:28.576421 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:28.576518 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:28.580712 (+ 4194us) server_negotiation.cc:299] Negotiation successful
0619 08:58:28.580767 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":58,"threads_started":1}
I20260619 08:58:28.581749 13559 tablet_service.cc:1467] Processing CreateTablet for tablet d0e8f8e39999403081bdae5cf5acff22 (DEFAULT_TABLE table=test-table [id=3dd9ea79795b44fc8ee26c9d4d2e9f52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:58:28.582060 13559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0e8f8e39999403081bdae5cf5acff22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:58:28.584935 13661 tablet_bootstrap.cc:492] T d0e8f8e39999403081bdae5cf5acff22 P 8bfb1ede8b9d4700ba75b36ff37f315e: Bootstrap starting.
I20260619 08:58:28.585635 13661 tablet_bootstrap.cc:654] T d0e8f8e39999403081bdae5cf5acff22 P 8bfb1ede8b9d4700ba75b36ff37f315e: Neither blocks nor log segments found. Creating new log.
I20260619 08:58:28.585949 13661 log.cc:824] T d0e8f8e39999403081bdae5cf5acff22 P 8bfb1ede8b9d4700ba75b36ff37f315e: Log is configured to *not* fsync() on all Append() calls
I20260619 08:58:28.586104 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:28.571596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39439 (local address 127.0.0.1:44866)
0619 08:58:28.573009 (+ 1413us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:28.573019 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:28.573064 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:28.574441 (+ 1377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:28.574442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:28.574444 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:58:28.574488 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:28.574489 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:28.578228 (+ 3739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:28.578229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:28.579373 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:28.579376 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:28.585774 (+ 6398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:28.585776 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:28.585855 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:28.585861 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 08:58:28.585905 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 08:58:28.585958 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1397,"thread_start_us":67,"threads_started":1}
I20260619 08:58:28.586437 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:28.572262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44866 (local address 127.0.98.2:39439)
0619 08:58:28.572403 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:28.572408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:28.573116 (+ 708us) server_negotiation.cc:407] Connection header received
0619 08:58:28.573151 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:28.573153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:28.573166 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:28.573224 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:58:28.576558 (+ 3334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:28.578140 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:28.579484 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:28.585556 (+ 6072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:28.585631 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:28.586283 (+ 652us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:28.586314 (+ 31us) server_negotiation.cc:299] Negotiation successful
0619 08:58:28.586363 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":54,"threads_started":1}
I20260619 08:58:28.586619 13660 tablet_bootstrap.cc:492] T d0e8f8e39999403081bdae5cf5acff22 P e0e9a356ca7f43b1ad934ed8fe981240: Bootstrap starting.
I20260619 08:58:28.587241 13660 tablet_bootstrap.cc:654] T d0e8f8e39999403081bdae5cf5acff22 P e0e9a356ca7f43b1ad934ed8fe981240: Neither blocks nor log segments found. Creating new log.
I20260619 08:58:28.587141 13427 tablet_service.cc:1467] Processing CreateTablet for tablet d0e8f8e39999403081bdae5cf5acff22 (DEFAULT_TABLE table=test-table [id=3dd9ea79795b44fc8ee26c9d4d2e9f52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:58:28.587463 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0e8f8e39999403081bdae5cf5acff22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:58:28.587503 13660 log.cc:824] T d0e8f8e39999403081bdae5cf5acff22 P e0e9a356ca7f43b1ad934ed8fe981240: Log is configured to *not* fsync() on all Append() calls
I20260619 08:58:28.588284 13661 tablet_bootstrap.cc:492] T d0e8f8e39999403081bdae5cf5acff22 P 8bfb1ede8b9d4700ba75b36ff37f315e: No bootstrap required, opened a new log
I20260619 08:58:28.588361 13661 ts_tablet_manager.cc:1397] T d0e8f8e39999403081bdae5cf5acff22 P 8bfb1ede8b9d4700ba75b36ff37f315e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 08:58:28.591463 13664 tablet_bootstrap.cc:492] T d0e8f8e39999403081bdae5cf5acff22 P 1355b420aa974adaa4762429a384db6c: Bootstrap starting.
I20260619 08:58:28.591379 13661 raft_consensus.cc:348] T d0e8f8e39999403081bdae5cf5acff22 P 8bfb1ede8b9d4700ba75b36ff37f315e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e9a356ca7f43b1ad934ed8fe981240" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37509 } } peers { permanent_uuid: "1355b420aa974adaa4762429a384db6c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39439 } } peers { permanent_uuid: "8bfb1ede8b9d4700ba75b36ff37f315e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35753 } }
I20260619 08:58:28.591540 13661 raft_consensus.cc:374] T d0e8f8e39999403081bdae5cf5acff22 P 8bfb1ede8b9d4700ba75b36ff37f315e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:58:28.591588 13661 raft_consensus.cc:729] T d0e8f8e39999403081bdae5cf5acff22 P 8bfb1ede8b9d4700ba75b36ff37f315e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bfb1ede8b9d4700ba75b36ff37f315e, State: Initialized, Role: FOLLOWER
I20260619 08:58:28.591694 13661 consensus_queue.cc:260] T d0e8f8e39999403081bdae5cf5acff22 P 8bfb1ede8b9d4700ba75b36ff37f315e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e9a356ca7f43b1ad934ed8fe981240" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37509 } } peers { permanent_uuid: "1355b420aa974adaa4762429a384db6c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39439 } } peers { permanent_uuid: "8bfb1ede8b9d4700ba75b36ff37f315e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35753 } }
I20260619 08:58:28.591955 13661 ts_tablet_manager.cc:1428] T d0e8f8e39999403081bdae5cf5acff22 P 8bfb1ede8b9d4700ba75b36ff37f315e: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260619 08:58:28.592432 13664 tablet_bootstrap.cc:654] T d0e8f8e39999403081bdae5cf5acff22 P 1355b420aa974adaa4762429a384db6c: Neither blocks nor log segments found. Creating new log.
I20260619 08:58:28.592514 13627 heartbeater.cc:503] Master 127.0.98.62:39541 was elected leader, sending a full tablet report...
I20260619 08:58:28.592810 13664 log.cc:824] T d0e8f8e39999403081bdae5cf5acff22 P 1355b420aa974adaa4762429a384db6c: Log is configured to *not* fsync() on all Append() calls
I20260619 08:58:28.593673 13660 tablet_bootstrap.cc:492] T d0e8f8e39999403081bdae5cf5acff22 P e0e9a356ca7f43b1ad934ed8fe981240: No bootstrap required, opened a new log
I20260619 08:58:28.593806 13660 ts_tablet_manager.cc:1397] T d0e8f8e39999403081bdae5cf5acff22 P e0e9a356ca7f43b1ad934ed8fe981240: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260619 08:58:28.594280 13664 tablet_bootstrap.cc:492] T d0e8f8e39999403081bdae5cf5acff22 P 1355b420aa974adaa4762429a384db6c: No bootstrap required, opened a new log
I20260619 08:58:28.594347 13664 ts_tablet_manager.cc:1397] T d0e8f8e39999403081bdae5cf5acff22 P 1355b420aa974adaa4762429a384db6c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 08:58:28.596630 13660 raft_consensus.cc:348] T d0e8f8e39999403081bdae5cf5acff22 P e0e9a356ca7f43b1ad934ed8fe981240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e9a356ca7f43b1ad934ed8fe981240" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37509 } } peers { permanent_uuid: "1355b420aa974adaa4762429a384db6c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39439 } } peers { permanent_uuid: "8bfb1ede8b9d4700ba75b36ff37f315e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35753 } }
I20260619 08:58:28.596786 13664 raft_consensus.cc:348] T d0e8f8e39999403081bdae5cf5acff22 P 1355b420aa974adaa4762429a384db6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e9a356ca7f43b1ad934ed8fe981240" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37509 } } peers { permanent_uuid: "1355b420aa974adaa4762429a384db6c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39439 } } peers { permanent_uuid: "8bfb1ede8b9d4700ba75b36ff37f315e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35753 } }
I20260619 08:58:28.596923 13664 raft_consensus.cc:374] T d0e8f8e39999403081bdae5cf5acff22 P 1355b420aa974adaa4762429a384db6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:58:28.596958 13664 raft_consensus.cc:729] T d0e8f8e39999403081bdae5cf5acff22 P 1355b420aa974adaa4762429a384db6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1355b420aa974adaa4762429a384db6c, State: Initialized, Role: FOLLOWER
I20260619 08:58:28.597087 13664 consensus_queue.cc:260] T d0e8f8e39999403081bdae5cf5acff22 P 1355b420aa974adaa4762429a384db6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e9a356ca7f43b1ad934ed8fe981240" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37509 } } peers { permanent_uuid: "1355b420aa974adaa4762429a384db6c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39439 } } peers { permanent_uuid: "8bfb1ede8b9d4700ba75b36ff37f315e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35753 } }
I20260619 08:58:28.596904 13660 raft_consensus.cc:374] T d0e8f8e39999403081bdae5cf5acff22 P e0e9a356ca7f43b1ad934ed8fe981240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:58:28.597265 13660 raft_consensus.cc:729] T d0e8f8e39999403081bdae5cf5acff22 P e0e9a356ca7f43b1ad934ed8fe981240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0e9a356ca7f43b1ad934ed8fe981240, State: Initialized, Role: FOLLOWER
I20260619 08:58:28.597434 13660 consensus_queue.cc:260] T d0e8f8e39999403081bdae5cf5acff22 P e0e9a356ca7f43b1ad934ed8fe981240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e9a356ca7f43b1ad934ed8fe981240" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37509 } } peers { permanent_uuid: "1355b420aa974adaa4762429a384db6c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39439 } } peers { permanent_uuid: "8bfb1ede8b9d4700ba75b36ff37f315e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35753 } }
I20260619 08:58:28.597688 13664 ts_tablet_manager.cc:1428] T d0e8f8e39999403081bdae5cf5acff22 P 1355b420aa974adaa4762429a384db6c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:58:28.597807 13493 heartbeater.cc:503] Master 127.0.98.62:39541 was elected leader, sending a full tablet report...
I20260619 08:58:28.597805 13660 ts_tablet_manager.cc:1428] T d0e8f8e39999403081bdae5cf5acff22 P e0e9a356ca7f43b1ad934ed8fe981240: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260619 08:58:28.597960 13359 heartbeater.cc:503] Master 127.0.98.62:39541 was elected leader, sending a full tablet report...
I20260619 08:58:28.651319 13665 raft_consensus.cc:482] T d0e8f8e39999403081bdae5cf5acff22 P 8bfb1ede8b9d4700ba75b36ff37f315e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:58:28.651463 13665 raft_consensus.cc:504] T d0e8f8e39999403081bdae5cf5acff22 P 8bfb1ede8b9d4700ba75b36ff37f315e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e9a356ca7f43b1ad934ed8fe981240" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37509 } } peers { permanent_uuid: "1355b420aa974adaa4762429a384db6c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39439 } } peers { permanent_uuid: "8bfb1ede8b9d4700ba75b36ff37f315e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35753 } }
I20260619 08:58:28.651855 13665 leader_election.cc:288] T d0e8f8e39999403081bdae5cf5acff22 P 8bfb1ede8b9d4700ba75b36ff37f315e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0e9a356ca7f43b1ad934ed8fe981240 (127.0.98.1:37509), 1355b420aa974adaa4762429a384db6c (127.0.98.2:39439)
I20260619 08:58:28.656673 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:28.652761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57395 (local address 127.0.98.1:37509)
0619 08:58:28.652805 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:28.652807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:28.653003 (+ 196us) server_negotiation.cc:407] Connection header received
0619 08:58:28.653111 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:28.653112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:28.653125 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:28.653154 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:58:28.653451 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:28.654172 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:28.655701 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:28.656307 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:28.656353 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:28.656363 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:28.656556 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 08:58:28.656596 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 08:58:28.657150 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:28.652307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37509 (local address 127.0.98.3:57395)
0619 08:58:28.652938 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:28.652949 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:28.653032 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:28.653230 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:28.653230 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:28.653232 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:58:28.653326 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:28.653328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:28.654291 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:28.654292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:28.655590 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:28.655594 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:28.656416 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:28.656417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:28.656486 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:28.656490 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:58:28.657006 (+ 516us) client_negotiation.cc:240] Negotiation successful
0619 08:58:28.657050 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":105,"threads_started":1}
I20260619 08:58:28.657636 13313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e8f8e39999403081bdae5cf5acff22" candidate_uuid: "8bfb1ede8b9d4700ba75b36ff37f315e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0e9a356ca7f43b1ad934ed8fe981240" is_pre_election: true
I20260619 08:58:28.657819 13313 raft_consensus.cc:2454] T d0e8f8e39999403081bdae5cf5acff22 P e0e9a356ca7f43b1ad934ed8fe981240 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bfb1ede8b9d4700ba75b36ff37f315e in term 0.
I20260619 08:58:28.657928 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:28.652851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45195 (local address 127.0.98.2:39439)
0619 08:58:28.652893 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:28.652895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:28.653041 (+ 146us) server_negotiation.cc:407] Connection header received
0619 08:58:28.653125 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:28.653126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:28.653137 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:28.653163 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:58:28.653404 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:28.654171 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:28.656997 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:28.657550 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:28.657591 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:28.657601 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:28.657821 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 08:58:28.657869 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 08:58:28.658147 13669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:28.652540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39439 (local address 127.0.98.3:45195)
0619 08:58:28.652991 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:28.652999 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:28.653059 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:28.653239 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:28.653240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:28.653241 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:58:28.653334 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:28.653335 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:28.655837 (+ 2502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:28.655838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:28.656906 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:28.656908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:28.657680 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:28.657682 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:28.657743 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:28.657746 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:58:28.658026 (+ 280us) client_negotiation.cc:240] Negotiation successful
0619 08:58:28.658068 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":25,"thread_start_us":60,"threads_started":1}
I20260619 08:58:28.658228 13514 leader_election.cc:302] T d0e8f8e39999403081bdae5cf5acff22 P 8bfb1ede8b9d4700ba75b36ff37f315e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bfb1ede8b9d4700ba75b36ff37f315e, e0e9a356ca7f43b1ad934ed8fe981240; no voters:
I20260619 08:58:28.658355 13665 raft_consensus.cc:2781] T d0e8f8e39999403081bdae5cf5acff22 P 8bfb1ede8b9d4700ba75b36ff37f315e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:58:28.658402 13665 raft_consensus.cc:482] T d0e8f8e39999403081bdae5cf5acff22 P 8bfb1ede8b9d4700ba75b36ff37f315e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:58:28.658424 13665 raft_consensus.cc:3037] T d0e8f8e39999403081bdae5cf5acff22 P 8bfb1ede8b9d4700ba75b36ff37f315e [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:58:28.658593 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e8f8e39999403081bdae5cf5acff22" candidate_uuid: "8bfb1ede8b9d4700ba75b36ff37f315e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1355b420aa974adaa4762429a384db6c" is_pre_election: true
I20260619 08:58:28.658773 13447 raft_consensus.cc:2454] T d0e8f8e39999403081bdae5cf5acff22 P 1355b420aa974adaa4762429a384db6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bfb1ede8b9d4700ba75b36ff37f315e in term 0.
I20260619 08:58:28.659301 13665 raft_consensus.cc:504] T d0e8f8e39999403081bdae5cf5acff22 P 8bfb1ede8b9d4700ba75b36ff37f315e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e9a356ca7f43b1ad934ed8fe981240" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37509 } } peers { permanent_uuid: "1355b420aa974adaa4762429a384db6c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39439 } } peers { permanent_uuid: "8bfb1ede8b9d4700ba75b36ff37f315e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35753 } }
I20260619 08:58:28.659433 13665 leader_election.cc:288] T d0e8f8e39999403081bdae5cf5acff22 P 8bfb1ede8b9d4700ba75b36ff37f315e [CANDIDATE]: Term 1 election: Requested vote from peers e0e9a356ca7f43b1ad934ed8fe981240 (127.0.98.1:37509), 1355b420aa974adaa4762429a384db6c (127.0.98.2:39439)
I20260619 08:58:28.659633 13313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e8f8e39999403081bdae5cf5acff22" candidate_uuid: "8bfb1ede8b9d4700ba75b36ff37f315e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0e9a356ca7f43b1ad934ed8fe981240"
I20260619 08:58:28.659718 13313 raft_consensus.cc:3037] T d0e8f8e39999403081bdae5cf5acff22 P e0e9a356ca7f43b1ad934ed8fe981240 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:58:28.660584 13313 raft_consensus.cc:2454] T d0e8f8e39999403081bdae5cf5acff22 P e0e9a356ca7f43b1ad934ed8fe981240 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bfb1ede8b9d4700ba75b36ff37f315e in term 1.
I20260619 08:58:28.660843 13514 leader_election.cc:302] T d0e8f8e39999403081bdae5cf5acff22 P 8bfb1ede8b9d4700ba75b36ff37f315e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bfb1ede8b9d4700ba75b36ff37f315e, e0e9a356ca7f43b1ad934ed8fe981240; no voters:
I20260619 08:58:28.660954 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e8f8e39999403081bdae5cf5acff22" candidate_uuid: "8bfb1ede8b9d4700ba75b36ff37f315e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1355b420aa974adaa4762429a384db6c"
I20260619 08:58:28.661028 13447 raft_consensus.cc:3037] T d0e8f8e39999403081bdae5cf5acff22 P 1355b420aa974adaa4762429a384db6c [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:58:28.661823 13447 raft_consensus.cc:2454] T d0e8f8e39999403081bdae5cf5acff22 P 1355b420aa974adaa4762429a384db6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bfb1ede8b9d4700ba75b36ff37f315e in term 1.
I20260619 08:58:28.662025 13665 raft_consensus.cc:2781] T d0e8f8e39999403081bdae5cf5acff22 P 8bfb1ede8b9d4700ba75b36ff37f315e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:58:28.662184 13665 raft_consensus.cc:686] T d0e8f8e39999403081bdae5cf5acff22 P 8bfb1ede8b9d4700ba75b36ff37f315e [term 1 LEADER]: Becoming Leader. State: Replica: 8bfb1ede8b9d4700ba75b36ff37f315e, State: Running, Role: LEADER
I20260619 08:58:28.662276 13665 consensus_queue.cc:237] T d0e8f8e39999403081bdae5cf5acff22 P 8bfb1ede8b9d4700ba75b36ff37f315e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e9a356ca7f43b1ad934ed8fe981240" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37509 } } peers { permanent_uuid: "1355b420aa974adaa4762429a384db6c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39439 } } peers { permanent_uuid: "8bfb1ede8b9d4700ba75b36ff37f315e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35753 } }
I20260619 08:58:28.663125 13161 catalog_manager.cc:5495] T d0e8f8e39999403081bdae5cf5acff22 P 8bfb1ede8b9d4700ba75b36ff37f315e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8bfb1ede8b9d4700ba75b36ff37f315e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8bfb1ede8b9d4700ba75b36ff37f315e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e9a356ca7f43b1ad934ed8fe981240" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1355b420aa974adaa4762429a384db6c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8bfb1ede8b9d4700ba75b36ff37f315e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35753 } health_report { overall_health: HEALTHY } } }
W20260619 08:58:28.669382 13360 tablet.cc:2369] T d0e8f8e39999403081bdae5cf5acff22 P e0e9a356ca7f43b1ad934ed8fe981240: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:58:28.673946 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:28.670873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42798 (local address 127.0.98.3:35753)
0619 08:58:28.670912 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:28.670914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:28.670922 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:58:28.671000 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:28.671001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:28.671022 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:28.671095 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:58:28.671242 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:28.671921 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:28.673012 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:28.673297 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:28.673368 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:28.673561 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:58:28.673696 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:58:28.673793 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:28.673845 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 08:58:28.673869 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 08:58:28.676255 13447 raft_consensus.cc:1261] T d0e8f8e39999403081bdae5cf5acff22 P 1355b420aa974adaa4762429a384db6c [term 1 FOLLOWER]: Refusing update from remote peer 8bfb1ede8b9d4700ba75b36ff37f315e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:58:28.676312 13313 raft_consensus.cc:1261] T d0e8f8e39999403081bdae5cf5acff22 P e0e9a356ca7f43b1ad934ed8fe981240 [term 1 FOLLOWER]: Refusing update from remote peer 8bfb1ede8b9d4700ba75b36ff37f315e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:58:28.676678 13671 consensus_queue.cc:1035] T d0e8f8e39999403081bdae5cf5acff22 P 8bfb1ede8b9d4700ba75b36ff37f315e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0e9a356ca7f43b1ad934ed8fe981240" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:58:28.676849 13671 consensus_queue.cc:1035] T d0e8f8e39999403081bdae5cf5acff22 P 8bfb1ede8b9d4700ba75b36ff37f315e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1355b420aa974adaa4762429a384db6c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:58:28.680025 13674 mvcc.cc:205] Tried to move back new op lower bound from 7298496547534458880 to 7298496547481161728. Current Snapshot: MvccSnapshot[applied={T|T < 7298496547534458880}]
I20260619 08:58:28.680687 13675 mvcc.cc:205] Tried to move back new op lower bound from 7298496547534458880 to 7298496547481161728. Current Snapshot: MvccSnapshot[applied={T|T < 7298496547534458880}]
I20260619 08:58:28.682999 13161 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:58:28.683101 13161 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:58:28.685199 13161 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b43e79008f147caa0502f3615f5ea8b: Sending DeleteTablet for 3 replicas of tablet d0e8f8e39999403081bdae5cf5acff22
I20260619 08:58:28.685709 13676 mvcc.cc:205] Tried to move back new op lower bound from 7298496547534458880 to 7298496547481161728. Current Snapshot: MvccSnapshot[applied={T|T < 7298496547534458880}]
I20260619 08:58:28.685963 13275 tablet_service.cc:1514] Processing DeleteTablet for tablet d0e8f8e39999403081bdae5cf5acff22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:58:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37390
I20260619 08:58:28.686216 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet d0e8f8e39999403081bdae5cf5acff22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:58:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44866
I20260619 08:58:28.686429 13559 tablet_service.cc:1514] Processing DeleteTablet for tablet d0e8f8e39999403081bdae5cf5acff22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:58:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42786
I20260619 08:58:28.686451 13684 tablet_replica.cc:330] T d0e8f8e39999403081bdae5cf5acff22 P 1355b420aa974adaa4762429a384db6c: stopping tablet replica
I20260619 08:58:28.686559 13684 raft_consensus.cc:2229] T d0e8f8e39999403081bdae5cf5acff22 P 1355b420aa974adaa4762429a384db6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:58:28.686661 13684 raft_consensus.cc:2258] T d0e8f8e39999403081bdae5cf5acff22 P 1355b420aa974adaa4762429a384db6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:58:28.687091 13684 ts_tablet_manager.cc:1905] T d0e8f8e39999403081bdae5cf5acff22 P 1355b420aa974adaa4762429a384db6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:58:28.687338 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 13199
I20260619 08:58:28.688992 13684 ts_tablet_manager.cc:1918] T d0e8f8e39999403081bdae5cf5acff22 P 1355b420aa974adaa4762429a384db6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:58:28.689050 13684 log.cc:1196] T d0e8f8e39999403081bdae5cf5acff22 P 1355b420aa974adaa4762429a384db6c: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-1/wal/wals/d0e8f8e39999403081bdae5cf5acff22
I20260619 08:58:28.689265 13684 ts_tablet_manager.cc:1939] T d0e8f8e39999403081bdae5cf5acff22 P 1355b420aa974adaa4762429a384db6c: Deleting consensus metadata
I20260619 08:58:28.689415 13685 tablet_replica.cc:330] T d0e8f8e39999403081bdae5cf5acff22 P 8bfb1ede8b9d4700ba75b36ff37f315e: stopping tablet replica
I20260619 08:58:28.689503 13685 raft_consensus.cc:2229] T d0e8f8e39999403081bdae5cf5acff22 P 8bfb1ede8b9d4700ba75b36ff37f315e [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:58:28.689595 13685 raft_consensus.cc:2258] T d0e8f8e39999403081bdae5cf5acff22 P 8bfb1ede8b9d4700ba75b36ff37f315e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:58:28.690232 13685 ts_tablet_manager.cc:1905] T d0e8f8e39999403081bdae5cf5acff22 P 8bfb1ede8b9d4700ba75b36ff37f315e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:58:28.690479 13145 catalog_manager.cc:4841] TS 1355b420aa974adaa4762429a384db6c (127.0.98.2:39439): tablet d0e8f8e39999403081bdae5cf5acff22 (table test-table [id=3dd9ea79795b44fc8ee26c9d4d2e9f52]) successfully deleted
I20260619 08:58:28.691633 13685 ts_tablet_manager.cc:1918] T d0e8f8e39999403081bdae5cf5acff22 P 8bfb1ede8b9d4700ba75b36ff37f315e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:58:28.691699 13685 log.cc:1196] T d0e8f8e39999403081bdae5cf5acff22 P 8bfb1ede8b9d4700ba75b36ff37f315e: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859361540516-392-0/minicluster-data/ts-2/wal/wals/d0e8f8e39999403081bdae5cf5acff22
I20260619 08:58:28.691947 13685 ts_tablet_manager.cc:1939] T d0e8f8e39999403081bdae5cf5acff22 P 8bfb1ede8b9d4700ba75b36ff37f315e: Deleting consensus metadata
I20260619 08:58:28.692319 13148 catalog_manager.cc:4841] TS 8bfb1ede8b9d4700ba75b36ff37f315e (127.0.98.3:35753): tablet d0e8f8e39999403081bdae5cf5acff22 (table test-table [id=3dd9ea79795b44fc8ee26c9d4d2e9f52]) successfully deleted
W20260619 08:58:28.703753 13145 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37509 (error 108)
W20260619 08:58:28.704174 13147 connection.cc:535] server connection from 127.0.98.1:60037 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 08:58:28.704831 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:28.704703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37509 (local address 127.0.0.1:37398)
0619 08:58:28.704761 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:28.704796 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260619 08:58:28.704967 13145 catalog_manager.cc:4584] TS e0e9a356ca7f43b1ad934ed8fe981240 (127.0.98.1:37509): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d0e8f8e39999403081bdae5cf5acff22: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37509: connect: Connection refused (error 111)
I20260619 08:58:28.705179 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 13363
I20260619 08:58:28.711935 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 13496
I20260619 08:58:28.718303 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 13128
I20260619 08:58:28.724030 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 08:58:29.058162 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 08:58:29.058247 392 mini_ranger.cc:66] Stopping Ranger...
I20260619 08:58:29.392388 392 mini_ranger.cc:68] Stopped Ranger
2026-06-19 08:58:29.392 UTC [12194] LOG: received smart shutdown request
2026-06-19 08:58:29.395 UTC [12194] LOG: background worker "logical replication launcher" (PID 12203) exited with exit code 1
2026-06-19 08:58:29.395 UTC [12198] LOG: shutting down
2026-06-19 08:58:29.538 UTC [12194] LOG: database system is shut down
2026-06-19T08:58:29Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46050 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:58:29 dist-test-slave-h4g5 krb5kdc[13696](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:58:29 dist-test-slave-h4g5 krb5kdc[13696](info): set up 2 sockets
Jun 19 08:58:29 dist-test-slave-h4g5 krb5kdc[13696](info): commencing operation
krb5kdc: starting...
W20260619 08:58:30.579774 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 08:58:30 dist-test-slave-h4g5 krb5kdc[13696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:58:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:58:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 08:58:30.706638 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41585
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41585,127.0.98.61:38621,127.0.98.60:43533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:58:30.785113 13709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:58:30.785310 13709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:58:30.785331 13709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:58:30.786396 13709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:58:30.786429 13709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:58:30.786441 13709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:58:30.786451 13709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:58:30.786461 13709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:58:30.787910 13709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41585,127.0.98.61:38621,127.0.98.60:43533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:58:30.788127 13709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:58:30.788311 13709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:58:30.792873 13715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:30.792884 13714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:30.792884 13717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:30.792945 13709 server_base.cc:1034] running on GCE node
I20260619 08:58:30.793301 13709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:58:30.793586 13709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:58:30.794740 13709 hybrid_clock.cc:648] HybridClock initialized: now 1781859510794727 us; error 32 us; skew 500 ppm
Jun 19 08:58:30 dist-test-slave-h4g5 krb5kdc[13696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:58:30.804987 13709 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 08:58:30.805397 13709 webserver.cc:460] Webserver started at http://127.0.98.62:37327/ using document root <none> and password file <none>
I20260619 08:58:30.805591 13709 fs_manager.cc:359] Metadata directory not provided
I20260619 08:58:30.805640 13709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:58:30.805764 13709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:58:30.806792 13709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "8647fa46d6bf408ca30acfc775dc14aa"
format_stamp: "Formatted at 2026-06-19 08:58:30 on dist-test-slave-h4g5"
server_key: "f3d764d78b5ba3a1596575eb261397f8"
server_key_iv: "11650a8db3f4511ebbddb8c6d5893951"
server_key_version: "encryptionkey@0"
I20260619 08:58:30.807123 13709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "8647fa46d6bf408ca30acfc775dc14aa"
format_stamp: "Formatted at 2026-06-19 08:58:30 on dist-test-slave-h4g5"
server_key: "f3d764d78b5ba3a1596575eb261397f8"
server_key_iv: "11650a8db3f4511ebbddb8c6d5893951"
server_key_version: "encryptionkey@0"
I20260619 08:58:30.808337 13709 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:58:30.808972 13724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:30.809116 13709 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:58:30.809196 13709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "8647fa46d6bf408ca30acfc775dc14aa"
format_stamp: "Formatted at 2026-06-19 08:58:30 on dist-test-slave-h4g5"
server_key: "f3d764d78b5ba3a1596575eb261397f8"
server_key_iv: "11650a8db3f4511ebbddb8c6d5893951"
server_key_version: "encryptionkey@0"
I20260619 08:58:30.809255 13709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:58:30.844527 13709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:58:30.846709 13709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:58:30.846850 13709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:58:30.850553 13709 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41585
I20260619 08:58:30.850593 13776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41585 every 8 connection(s)
I20260619 08:58:30.850903 13709 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:58:30.851222 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 13709
I20260619 08:58:30.851308 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 08:58:30.851486 392 external_mini_cluster.cc:1428] Setting key d9fd4efda171898b734f5fc10c39bdd2
I20260619 08:58:30.851751 13777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:58:30.853315 13777 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41585 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 19 08:58:30 dist-test-slave-h4g5 krb5kdc[13696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:58:30.864790 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:30.854081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41585 (local address 127.0.0.1:55324)
0619 08:58:30.854250 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:30.854290 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:30.854874 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:30.855144 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:30.855147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:30.855471 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:58:30.855705 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:30.855711 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:30.856731 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:30.856733 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:30.857800 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:30.857803 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:30.858253 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:30.858254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:30.858335 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:30.858854 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:58:30.858865 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:58:30.861870 (+ 3005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:58:30.863393 (+ 1523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:30.863399 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:30.863401 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:30.863606 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:30.863815 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:30.863815 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:30.863816 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:30.863887 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:30.864243 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:58:30.864248 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:58:30.864312 (+ 64us) client_negotiation.cc:769] Sending connection context
0619 08:58:30.864389 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 08:58:30.864600 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":82,"thread_start_us":84,"threads_started":1}
I20260619 08:58:30.864790 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:30.854119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55324 (local address 127.0.98.62:41585)
0619 08:58:30.854258 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:30.854268 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:30.854379 (+ 111us) server_negotiation.cc:407] Connection header received
0619 08:58:30.855022 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:30.855024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:30.855040 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:30.855112 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:30.855786 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:30.856647 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:30.857877 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:30.858173 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:30.858246 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:30.861949 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:30.861955 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:30.861956 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:30.861963 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:30.863323 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:30.863680 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:30.863683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:30.863684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:30.863758 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:30.863940 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:30.863941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:30.863941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:30.864186 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:30.864259 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:30.864416 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 08:58:30.864546 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
I20260619 08:58:30.865478 13777 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38621 } has no permanent_uuid. Determining permanent_uuid...
I20260619 08:58:30.865782 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:30.865679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38621 (local address 127.0.0.1:46976)
0619 08:58:30.865718 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:30.865761 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260619 08:58:30.865942 13728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38621: connect: Connection refused (error 111)
I20260619 08:58:30.866348 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:30.866269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38621 (local address 127.0.0.1:46982)
0619 08:58:30.866309 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:30.866331 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 08:58:30.866441 13777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38621: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38621: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 08:58:30.875052 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38621
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41585,127.0.98.61:38621,127.0.98.60:43533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 08:58:30.915663 13777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38621 } attempt: 1
I20260619 08:58:30.916179 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:30.916029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38621 (local address 127.0.0.1:46992)
0619 08:58:30.916110 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:30.916146 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 08:58:30.916532 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:30.916449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38621 (local address 127.0.0.1:47008)
0619 08:58:30.916490 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:30.916515 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 08:58:30.916630 13777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38621: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38621: connect: Connection refused (error 111)
W20260619 08:58:30.953011 13784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:58:30.953222 13784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:58:30.953253 13784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:58:30.954419 13784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:58:30.954461 13784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:58:30.954480 13784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:58:30.954495 13784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:58:30.954509 13784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:58:30.956013 13784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41585,127.0.98.61:38621,127.0.98.60:43533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:58:30.956284 13784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:58:30.956502 13784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:58:30.961028 13792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:30.961112 13790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:30.961086 13789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:30.961249 13784 server_base.cc:1034] running on GCE node
I20260619 08:58:30.961462 13784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:58:30.961709 13784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:58:30.962843 13784 hybrid_clock.cc:648] HybridClock initialized: now 1781859510962827 us; error 29 us; skew 500 ppm
Jun 19 08:58:30 dist-test-slave-h4g5 krb5kdc[13696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:58:30.965196 13784 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 08:58:30.965576 13784 webserver.cc:460] Webserver started at http://127.0.98.61:42227/ using document root <none> and password file <none>
I20260619 08:58:30.965741 13784 fs_manager.cc:359] Metadata directory not provided
I20260619 08:58:30.965778 13784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:58:30.965868 13784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:58:30.966863 13784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-1/data/instance:
uuid: "96ea9a7b62c04a32b67707dc093f4d50"
format_stamp: "Formatted at 2026-06-19 08:58:30 on dist-test-slave-h4g5"
server_key: "0471804ad88ff7209b9f2e9aabe99a9a"
server_key_iv: "b0d195b06c0f213956e7b93b5f97faf1"
server_key_version: "encryptionkey@0"
I20260619 08:58:30.967159 13784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-1/wal/instance:
uuid: "96ea9a7b62c04a32b67707dc093f4d50"
format_stamp: "Formatted at 2026-06-19 08:58:30 on dist-test-slave-h4g5"
server_key: "0471804ad88ff7209b9f2e9aabe99a9a"
server_key_iv: "b0d195b06c0f213956e7b93b5f97faf1"
server_key_version: "encryptionkey@0"
I20260619 08:58:30.968348 13784 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:58:30.969107 13799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:30.969280 13784 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:58:30.969359 13784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-1/wal
uuid: "96ea9a7b62c04a32b67707dc093f4d50"
format_stamp: "Formatted at 2026-06-19 08:58:30 on dist-test-slave-h4g5"
server_key: "0471804ad88ff7209b9f2e9aabe99a9a"
server_key_iv: "b0d195b06c0f213956e7b93b5f97faf1"
server_key_version: "encryptionkey@0"
I20260619 08:58:30.969414 13784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:58:30.984768 13777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38621 } attempt: 2
I20260619 08:58:30.985185 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:30.985071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38621 (local address 127.0.0.1:47022)
0619 08:58:30.985128 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:30.985157 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 08:58:30.985525 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:30.985444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38621 (local address 127.0.0.1:47032)
0619 08:58:30.985484 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:30.985508 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 08:58:30.985608 13777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38621: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38621: connect: Connection refused (error 111)
I20260619 08:58:30.986295 13784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:58:30.988113 13784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:58:30.988256 13784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:58:30.991832 13784 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:38621
I20260619 08:58:30.991880 13851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:38621 every 8 connection(s)
I20260619 08:58:30.992254 13784 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-1/data/info.pb
I20260619 08:58:30.993028 13852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:58:30.994438 13852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41585 } has no permanent_uuid. Determining permanent_uuid...
I20260619 08:58:30.999509 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 13784
I20260619 08:58:30.999603 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-1/wal/instance
I20260619 08:58:30.999765 392 external_mini_cluster.cc:1428] Setting key 2e5baa60f2a5dd0ab1b504b081c3b0b0
Jun 19 08:58:30 dist-test-slave-h4g5 krb5kdc[13696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:58:31.002960 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:30.995056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55330 (local address 127.0.98.62:41585)
0619 08:58:30.995113 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:30.995115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:30.995301 (+ 186us) server_negotiation.cc:407] Connection header received
0619 08:58:30.995744 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:30.995745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:30.995757 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:30.995798 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:30.996395 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:30.996894 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:30.997947 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:30.998173 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:30.998235 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:31.000325 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:31.000327 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:31.000327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:31.000335 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:31.001651 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:31.002056 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:31.002057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:31.002058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:31.002126 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:31.002340 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:31.002341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:31.002341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:31.002480 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:31.002542 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:31.002771 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 08:58:31.002861 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 08:58:31.003157 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:30.995045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41585 (local address 127.0.0.1:55330)
0619 08:58:30.995193 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:30.995227 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:30.995657 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:30.995848 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:30.995851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:30.996101 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:58:30.996302 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:30.996307 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:30.996995 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:30.996996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:30.997844 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:30.997847 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:30.998266 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:30.998267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:30.998379 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:30.998857 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:58:30.998866 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:58:31.000235 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:58:31.001727 (+ 1492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:31.001734 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:31.001735 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:31.001973 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:31.002186 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:31.002187 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:31.002187 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:31.002286 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:31.002534 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:58:31.002537 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:58:31.002654 (+ 117us) client_negotiation.cc:769] Sending connection context
0619 08:58:31.002732 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 08:58:31.002857 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":81,"threads_started":1}
I20260619 08:58:31.003691 13852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38621 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 19 08:58:31 dist-test-slave-h4g5 krb5kdc[13696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 08:58:31.013377 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:31.003955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38621 (local address 127.0.0.1:47048)
0619 08:58:31.003996 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:31.004006 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:31.004043 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:31.004310 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:31.004311 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:31.004420 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:58:31.004497 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:31.004499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.005748 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.005749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:31.006248 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:31.006250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.006860 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.006861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:31.006911 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:31.007319 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:58:31.007323 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:58:31.010386 (+ 3063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:58:31.012304 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:31.012305 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:31.012305 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:31.012459 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:31.012646 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:31.012647 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:31.012647 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:31.012690 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:31.013062 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:58:31.013063 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:58:31.013104 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 08:58:31.013182 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 08:58:31.013304 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260619 08:58:31.013516 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:31.003991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47048 (local address 127.0.98.61:38621)
0619 08:58:31.004161 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:31.004167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:31.004190 (+ 23us) server_negotiation.cc:407] Connection header received
0619 08:58:31.004212 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:31.004214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:31.004229 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:31.004310 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:31.004565 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.005672 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.006325 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.006793 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.006866 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:31.010474 (+ 3608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:31.010483 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:31.010483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:31.010490 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:31.012234 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:31.012528 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:31.012532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:31.012532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:31.012592 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:31.012738 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:31.012739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:31.012739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:31.012965 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:31.013046 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:31.013228 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 08:58:31.013369 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":140,"thread_start_us":109,"threads_started":1}
I20260619 08:58:31.014086 13852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43533 } has no permanent_uuid. Determining permanent_uuid...
I20260619 08:58:31.014425 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:31.014315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43533 (local address 127.0.0.1:47016)
0619 08:58:31.014358 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:31.014403 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 08:58:31.014566 13800 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43533: connect: Connection refused (error 111)
I20260619 08:58:31.014971 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:31.014892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43533 (local address 127.0.0.1:47022)
0619 08:58:31.014931 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:31.014954 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260619 08:58:31.015067 13852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43533: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43533: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 08:58:31.025185 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43533
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41585,127.0.98.61:38621,127.0.98.60:43533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 08:58:31.064293 13852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43533 } attempt: 1
I20260619 08:58:31.064769 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:31.064611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43533 (local address 127.0.0.1:47034)
0619 08:58:31.064674 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:31.064725 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 08:58:31.065137 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:31.065053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43533 (local address 127.0.0.1:47040)
0619 08:58:31.065088 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:31.065115 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 08:58:31.065229 13852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43533: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43533: connect: Connection refused (error 111)
I20260619 08:58:31.076749 13777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38621 } attempt: 3
Jun 19 08:58:31 dist-test-slave-h4g5 krb5kdc[13696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 08:58:31.083245 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:31.077102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38621 (local address 127.0.0.1:47052)
0619 08:58:31.077155 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:31.077166 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:31.077213 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:31.077338 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:31.077339 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:31.077505 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:58:31.077566 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:31.077568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.078227 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.078229 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:31.078726 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:31.078727 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.079077 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.079078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:31.079149 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:31.079607 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:58:31.079611 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:58:31.080942 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:58:31.082337 (+ 1395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:31.082338 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:31.082339 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:31.082506 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:31.082667 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:31.082668 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:31.082668 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:31.082723 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:31.082929 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:58:31.082930 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:58:31.082978 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 08:58:31.083039 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 08:58:31.083142 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 08:58:31.083245 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:31.077172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47052 (local address 127.0.98.61:38621)
0619 08:58:31.077232 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:31.077234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:31.077254 (+ 20us) server_negotiation.cc:407] Connection header received
0619 08:58:31.077279 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:31.077280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:31.077288 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:31.077328 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:31.077638 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.078150 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.078796 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.079014 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.079071 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:31.081024 (+ 1953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:31.081025 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:31.081025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:31.081033 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:31.082264 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:31.082566 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:31.082567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:31.082567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:31.082616 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:31.082769 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:31.082770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:31.082770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:31.082876 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:31.082938 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:31.083064 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 08:58:31.083142 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 08:58:31.083648 13777 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43533 } has no permanent_uuid. Determining permanent_uuid...
I20260619 08:58:31.083992 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:31.083894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43533 (local address 127.0.0.1:47054)
0619 08:58:31.083938 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:31.083970 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 08:58:31.084319 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:31.084240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43533 (local address 127.0.0.1:47068)
0619 08:58:31.084276 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:31.084298 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 08:58:31.084399 13777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43533: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43533: connect: Connection refused (error 111)
W20260619 08:58:31.107080 13859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:58:31.107316 13859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:58:31.107349 13859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:58:31.108530 13859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:58:31.108577 13859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:58:31.108596 13859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:58:31.108611 13859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:58:31.108628 13859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:58:31.110229 13859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41585,127.0.98.61:38621,127.0.98.60:43533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:58:31.110495 13859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:58:31.110702 13859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:58:31.115381 13864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:31.115532 13777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43533 } attempt: 1
I20260619 08:58:31.115640 13859 server_base.cc:1034] running on GCE node
W20260619 08:58:31.115384 13865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:31.115424 13867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:31.115932 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:31.115808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43533 (local address 127.0.0.1:47074)
0619 08:58:31.115864 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:31.115896 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 08:58:31.116168 13859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:58:31.116276 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:31.116203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43533 (local address 127.0.0.1:47080)
0619 08:58:31.116229 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:31.116255 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260619 08:58:31.116345 13777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43533: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43533: connect: Connection refused (error 111)
I20260619 08:58:31.116500 13859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:58:31.117641 13859 hybrid_clock.cc:648] HybridClock initialized: now 1781859511117629 us; error 26 us; skew 500 ppm
Jun 19 08:58:31 dist-test-slave-h4g5 krb5kdc[13696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:58:31.120061 13859 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 08:58:31.120517 13859 webserver.cc:460] Webserver started at http://127.0.98.60:46875/ using document root <none> and password file <none>
I20260619 08:58:31.120718 13859 fs_manager.cc:359] Metadata directory not provided
I20260619 08:58:31.120760 13859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:58:31.120880 13859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:58:31.121907 13859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-2/data/instance:
uuid: "125f45ea6d3b4b338ded928eb53369f9"
format_stamp: "Formatted at 2026-06-19 08:58:31 on dist-test-slave-h4g5"
server_key: "314f281b440ffcdb6513438cbcad9574"
server_key_iv: "ab1c942d50ba58d56b90894ddcf2ba58"
server_key_version: "encryptionkey@0"
I20260619 08:58:31.122239 13859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-2/wal/instance:
uuid: "125f45ea6d3b4b338ded928eb53369f9"
format_stamp: "Formatted at 2026-06-19 08:58:31 on dist-test-slave-h4g5"
server_key: "314f281b440ffcdb6513438cbcad9574"
server_key_iv: "ab1c942d50ba58d56b90894ddcf2ba58"
server_key_version: "encryptionkey@0"
I20260619 08:58:31.123401 13859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 08:58:31.124049 13874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:31.124223 13859 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:58:31.124310 13859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-2/wal
uuid: "125f45ea6d3b4b338ded928eb53369f9"
format_stamp: "Formatted at 2026-06-19 08:58:31 on dist-test-slave-h4g5"
server_key: "314f281b440ffcdb6513438cbcad9574"
server_key_iv: "ab1c942d50ba58d56b90894ddcf2ba58"
server_key_version: "encryptionkey@0"
I20260619 08:58:31.124364 13859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:58:31.133386 13852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43533 } attempt: 2
I20260619 08:58:31.133625 13859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:58:31.133816 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:31.133697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43533 (local address 127.0.0.1:47084)
0619 08:58:31.133744 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:31.133783 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 08:58:31.134181 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:31.134089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43533 (local address 127.0.0.1:47090)
0619 08:58:31.134130 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:31.134156 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260619 08:58:31.134269 13852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43533: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43533: connect: Connection refused (error 111)
I20260619 08:58:31.135524 13859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:58:31.135650 13859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:58:31.139226 13859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43533
I20260619 08:58:31.139264 13926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43533 every 8 connection(s)
I20260619 08:58:31.139568 13859 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-2/data/info.pb
I20260619 08:58:31.140398 13927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:58:31.141863 13927 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41585 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 08:58:31 dist-test-slave-h4g5 krb5kdc[13696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:58:31.149677 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:31.142516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55340 (local address 127.0.98.62:41585)
0619 08:58:31.142575 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:31.142577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:31.142749 (+ 172us) server_negotiation.cc:407] Connection header received
0619 08:58:31.143183 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:31.143184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:31.143193 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:31.143236 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:31.143796 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.144303 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.145210 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.145384 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.145433 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:31.147329 (+ 1896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:31.147330 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:31.147331 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:31.147337 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:31.148596 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:31.148901 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:31.148902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:31.148902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:31.148943 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:31.149129 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:31.149129 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:31.149130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:31.149227 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:31.149306 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:31.149521 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 08:58:31.149586 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 08:58:31.149750 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 13859
I20260619 08:58:31.149761 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:31.142493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41585 (local address 127.0.0.1:55340)
0619 08:58:31.142645 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:31.142679 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:31.143110 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:31.143296 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:31.143299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:31.143543 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:58:31.143734 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:31.143738 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.144365 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.144366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:31.145144 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:31.145146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.145438 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.145438 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:31.145537 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:31.145989 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:58:31.145997 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:58:31.147254 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:58:31.148652 (+ 1398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:31.148655 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:31.148657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:31.148850 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:31.148995 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:31.148995 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:31.148996 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:31.149085 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:31.149299 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:58:31.149301 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:58:31.149406 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 08:58:31.149491 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 08:58:31.149601 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260619 08:58:31.149829 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859361540516-392-0/minicluster-data/master-2/wal/instance
I20260619 08:58:31.150009 392 external_mini_cluster.cc:1428] Setting key 1b6502316e25d6f14f3969a69687bf5e
I20260619 08:58:31.150274 13927 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38621 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 08:58:31 dist-test-slave-h4g5 krb5kdc[13696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 08:58:31 dist-test-slave-h4g5 krb5kdc[13696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 08:58:31.157073 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:31.150492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38621 (local address 127.0.0.1:47060)
0619 08:58:31.150536 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:31.150545 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:31.150585 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:31.150782 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:31.150782 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:31.150886 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:58:31.150926 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:31.150927 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.151648 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.151649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:31.152431 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:31.152433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.152837 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.152837 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:31.152893 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:31.153274 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:58:31.153279 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:58:31.154631 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:58:31.156142 (+ 1511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:31.156143 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:31.156143 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:31.156345 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:31.156521 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:31.156522 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:31.156522 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:31.156583 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:31.156797 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:58:31.156798 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:58:31.156842 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 08:58:31.156897 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 08:58:31.156977 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 08:58:31.157089 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:31.150589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47060 (local address 127.0.98.61:38621)
0619 08:58:31.150634 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:31.150636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:31.150643 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:58:31.150715 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:31.150717 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:31.150727 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:31.150780 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:31.150995 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.151570 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.152492 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.152764 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.152833 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:31.154712 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:31.154713 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:31.154714 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:31.154722 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:31.156048 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:31.156402 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:31.156403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:31.156403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:31.156464 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:31.156645 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:31.156646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:31.156646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:31.156744 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:31.156789 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:31.156935 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 08:58:31.157027 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 08:58:31.157493 13927 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43533 } has no permanent_uuid. Determining permanent_uuid...
I20260619 08:58:31.158643 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:31.150328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47104 (local address 127.0.98.60:43533)
0619 08:58:31.150490 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:31.150493 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:31.150590 (+ 97us) server_negotiation.cc:407] Connection header received
0619 08:58:31.150611 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:31.150613 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:31.150625 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:31.150698 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:31.151341 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.152394 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.153296 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.153733 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.153792 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:31.156061 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:31.156068 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:31.156068 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:31.156073 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:31.157620 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:31.157944 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:31.157947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:31.157948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:31.158005 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:31.158168 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:31.158169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:31.158169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:31.158326 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:31.158443 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:31.158528 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 08:58:31.158580 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":80,"threads_started":1}
Jun 19 08:58:31 dist-test-slave-h4g5 krb5kdc[13696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 08:58:31 dist-test-slave-h4g5 krb5kdc[13696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:58:31.164382 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:31.157744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43533 (local address 127.0.0.1:47106)
0619 08:58:31.157886 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:31.157895 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:31.157931 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:31.158041 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:31.158041 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:31.158157 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:58:31.158190 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:31.158192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.159017 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.159017 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:31.159667 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:31.159670 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.160034 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.160035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:31.160110 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:31.160453 (+ 343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:58:31.160457 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:58:31.161466 (+ 1009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:58:31.163341 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:31.163342 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:31.163342 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:31.163516 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:31.163724 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:31.163724 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:31.163725 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:31.163779 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:31.164038 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:58:31.164038 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:58:31.164110 (+ 72us) client_negotiation.cc:769] Sending connection context
0619 08:58:31.164174 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 08:58:31.164262 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260619 08:58:31.164402 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:31.157781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47106 (local address 127.0.98.60:43533)
0619 08:58:31.157928 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:31.157930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:31.157937 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:58:31.157977 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:31.157978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:31.157988 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:31.158036 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:31.158251 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.158926 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.159754 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.159968 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.160019 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:31.161531 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:31.161532 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:31.161532 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:31.161539 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:31.163257 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:31.163585 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:31.163586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:31.163586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:31.163642 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:31.163853 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:31.163854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:31.163855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:31.163969 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:31.164033 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:31.164209 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 08:58:31.164321 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
I20260619 08:58:31.166117 13927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 125f45ea6d3b4b338ded928eb53369f9: Bootstrap starting.
I20260619 08:58:31.166671 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:31.159434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55342 (local address 127.0.98.62:41585)
0619 08:58:31.159468 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:31.159469 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:31.159477 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:58:31.159530 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:31.159530 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:31.159537 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:31.159576 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:31.160061 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.160648 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.161355 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.161546 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.161613 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:31.163758 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:31.163759 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:31.163760 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:31.163766 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:31.165260 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:31.165541 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:31.165542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:31.165543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:31.165598 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:31.165734 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:31.165734 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:31.165735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:31.165854 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:31.166466 (+ 612us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:31.166511 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 08:58:31.166592 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 08:58:31.166882 13927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 125f45ea6d3b4b338ded928eb53369f9: Neither blocks nor log segments found. Creating new log.
I20260619 08:58:31.167171 13927 log.cc:824] T 00000000000000000000000000000000 P 125f45ea6d3b4b338ded928eb53369f9: Log is configured to *not* fsync() on all Append() calls
I20260619 08:58:31.167970 13927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 125f45ea6d3b4b338ded928eb53369f9: No bootstrap required, opened a new log
I20260619 08:58:31.169765 13927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 125f45ea6d3b4b338ded928eb53369f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8647fa46d6bf408ca30acfc775dc14aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41585 } } peers { permanent_uuid: "96ea9a7b62c04a32b67707dc093f4d50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38621 } } peers { permanent_uuid: "125f45ea6d3b4b338ded928eb53369f9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43533 } }
I20260619 08:58:31.169911 13927 raft_consensus.cc:374] T 00000000000000000000000000000000 P 125f45ea6d3b4b338ded928eb53369f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:58:31.169945 13927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 125f45ea6d3b4b338ded928eb53369f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125f45ea6d3b4b338ded928eb53369f9, State: Initialized, Role: FOLLOWER
I20260619 08:58:31.170092 13927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 125f45ea6d3b4b338ded928eb53369f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8647fa46d6bf408ca30acfc775dc14aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41585 } } peers { permanent_uuid: "96ea9a7b62c04a32b67707dc093f4d50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38621 } } peers { permanent_uuid: "125f45ea6d3b4b338ded928eb53369f9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43533 } }
I20260619 08:58:31.170352 13934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 125f45ea6d3b4b338ded928eb53369f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8647fa46d6bf408ca30acfc775dc14aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41585 } } peers { permanent_uuid: "96ea9a7b62c04a32b67707dc093f4d50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38621 } } peers { permanent_uuid: "125f45ea6d3b4b338ded928eb53369f9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43533 } } }
I20260619 08:58:31.170428 13934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 125f45ea6d3b4b338ded928eb53369f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 08:58:31.170439 13927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 125f45ea6d3b4b338ded928eb53369f9 [sys.catalog]: configured and running, proceeding with master startup.
Jun 19 08:58:31 dist-test-slave-h4g5 krb5kdc[13696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
W20260619 08:58:31.172211 13945 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 125f45ea6d3b4b338ded928eb53369f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 08:58:31.172266 13945 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 08:58:31.172856 13859 master_runner.cc:186] Error getting master registration for 127.0.98.62:41585: OK, instance_id { permanent_uuid: "8647fa46d6bf408ca30acfc775dc14aa" instance_seqno: 1781859510849505 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 08:58:31.174448 13859 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 08:58:31.174849 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:31.167237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47068 (local address 127.0.98.61:38621)
0619 08:58:31.167276 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:31.167278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:31.167354 (+ 76us) server_negotiation.cc:407] Connection header received
0619 08:58:31.167414 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:31.167415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:31.167424 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:31.167477 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:31.167950 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.168449 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.169318 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.169489 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.169531 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:31.171691 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:31.171693 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:31.171693 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:31.171701 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:31.173365 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:31.173732 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:31.173733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:31.173733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:31.173789 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:31.173980 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:31.173980 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:31.173981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:31.174101 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:31.174646 (+ 545us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:31.174687 (+ 41us) server_negotiation.cc:299] Negotiation successful
0619 08:58:31.174759 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 08:58:31.176508 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 08:58:31.183173 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:31.177671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47118 (local address 127.0.98.60:43533)
0619 08:58:31.177713 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:31.177715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:31.177976 (+ 261us) server_negotiation.cc:407] Connection header received
0619 08:58:31.177998 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:31.177998 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:31.178007 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:31.178047 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:31.178538 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.179014 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.179669 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.179876 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.179927 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:31.180965 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:31.180972 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:31.180973 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:31.180980 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:31.182158 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:31.182447 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:31.182448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:31.182448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:31.182499 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:31.182655 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:31.182656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:31.182656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:31.182796 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:31.182882 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:31.183015 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 08:58:31.183089 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 08:58:31.189543 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:31.177631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55344 (local address 127.0.98.62:41585)
0619 08:58:31.177668 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:31.177670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:31.183182 (+ 5512us) server_negotiation.cc:407] Connection header received
0619 08:58:31.183633 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:31.183634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:31.183645 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:31.183673 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:31.184211 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.184657 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.185465 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.185770 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.185836 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:31.186892 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:31.186893 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:31.186893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:31.186900 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:31.188382 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:31.188805 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:31.188805 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:31.188806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:31.188861 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:31.189036 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:31.189037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:31.189037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:31.189170 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:31.189253 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:31.189378 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 08:58:31.189455 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 08:58:31.189620 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:31.177942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47080 (local address 127.0.98.61:38621)
0619 08:58:31.177977 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:31.177979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:31.183344 (+ 5365us) server_negotiation.cc:407] Connection header received
0619 08:58:31.183435 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:31.183436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:31.183456 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:31.183484 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:31.184168 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.184892 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.185674 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.185895 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.185958 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:31.187168 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:31.187169 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:31.187169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:31.187176 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:31.188414 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:31.188944 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:31.188945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:31.188945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:31.188997 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:31.189158 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:31.189159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:31.189160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:31.189277 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:31.189366 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:31.189453 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 08:58:31.189528 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 08:58:31.191490 13777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43533 } attempt: 2
Jun 19 08:58:31 dist-test-slave-h4g5 krb5kdc[13696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 08:58:31.197997 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:31.191800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47132 (local address 127.0.98.60:43533)
0619 08:58:31.191844 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:31.191846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:31.191871 (+ 25us) server_negotiation.cc:407] Connection header received
0619 08:58:31.191892 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:31.191893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:31.191902 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:31.191954 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:31.192228 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.192953 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.193556 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.193849 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.193912 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:31.195489 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:31.195491 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:31.195491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:31.195497 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:31.196917 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:31.197313 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:31.197314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:31.197315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:31.197362 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:31.197566 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:31.197567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:31.197567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:31.197663 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:31.197710 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:31.197860 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 08:58:31.197914 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 08:58:31.198046 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:31.191737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43533 (local address 127.0.0.1:47132)
0619 08:58:31.191788 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:31.191798 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:31.191840 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:31.191955 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:31.191956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:31.192130 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:58:31.192170 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:31.192171 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.193021 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.193022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:31.193493 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:31.193495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.193906 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.193906 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:31.193939 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:31.194271 (+ 332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:58:31.194276 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:58:31.195397 (+ 1121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:58:31.196996 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:31.196997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:31.196998 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:31.197239 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:31.197419 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:31.197420 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:31.197420 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:31.197510 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:31.197717 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:58:31.197718 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:58:31.197771 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 08:58:31.197845 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 08:58:31.197946 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 08:58:31.199640 13777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8647fa46d6bf408ca30acfc775dc14aa: Bootstrap starting.
I20260619 08:58:31.200383 13777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8647fa46d6bf408ca30acfc775dc14aa: Neither blocks nor log segments found. Creating new log.
I20260619 08:58:31.200685 13777 log.cc:824] T 00000000000000000000000000000000 P 8647fa46d6bf408ca30acfc775dc14aa: Log is configured to *not* fsync() on all Append() calls
I20260619 08:58:31.201493 13777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8647fa46d6bf408ca30acfc775dc14aa: No bootstrap required, opened a new log
I20260619 08:58:31.203454 13777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8647fa46d6bf408ca30acfc775dc14aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8647fa46d6bf408ca30acfc775dc14aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41585 } } peers { permanent_uuid: "96ea9a7b62c04a32b67707dc093f4d50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38621 } } peers { permanent_uuid: "125f45ea6d3b4b338ded928eb53369f9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43533 } }
I20260619 08:58:31.203613 13777 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8647fa46d6bf408ca30acfc775dc14aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:58:31.203655 13777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8647fa46d6bf408ca30acfc775dc14aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8647fa46d6bf408ca30acfc775dc14aa, State: Initialized, Role: FOLLOWER
I20260619 08:58:31.203774 13777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8647fa46d6bf408ca30acfc775dc14aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8647fa46d6bf408ca30acfc775dc14aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41585 } } peers { permanent_uuid: "96ea9a7b62c04a32b67707dc093f4d50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38621 } } peers { permanent_uuid: "125f45ea6d3b4b338ded928eb53369f9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43533 } }
I20260619 08:58:31.204037 13955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8647fa46d6bf408ca30acfc775dc14aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8647fa46d6bf408ca30acfc775dc14aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41585 } } peers { permanent_uuid: "96ea9a7b62c04a32b67707dc093f4d50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38621 } } peers { permanent_uuid: "125f45ea6d3b4b338ded928eb53369f9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43533 } } }
I20260619 08:58:31.204114 13777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8647fa46d6bf408ca30acfc775dc14aa [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:58:31.204161 13955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8647fa46d6bf408ca30acfc775dc14aa [sys.catalog]: This master's current role is: FOLLOWER
W20260619 08:58:31.205825 13966 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8647fa46d6bf408ca30acfc775dc14aa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 08:58:31.205948 13966 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 08:58:31.206580 13709 master_runner.cc:186] Error getting master registration for 127.0.98.61:38621: OK, instance_id { permanent_uuid: "96ea9a7b62c04a32b67707dc093f4d50" instance_seqno: 1781859510990844 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 08:58:31.209316 13709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 08:58:31.225401 13852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43533 } attempt: 3
Jun 19 08:58:31 dist-test-slave-h4g5 krb5kdc[13696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 08:58:31.233405 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:31.225731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47146 (local address 127.0.98.60:43533)
0619 08:58:31.225777 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:31.225779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:31.225804 (+ 25us) server_negotiation.cc:407] Connection header received
0619 08:58:31.225824 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:31.225825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:31.225834 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:31.225885 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:31.226211 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.226999 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.228067 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.228387 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.228470 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:31.230663 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:31.230665 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:31.230665 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:31.230673 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:31.232244 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:31.232622 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:31.232623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:31.232623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:31.232680 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:31.232886 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:31.232888 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:31.232888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:31.233029 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:31.233096 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:31.233236 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 08:58:31.233312 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 08:58:31.233405 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:31.225664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43533 (local address 127.0.0.1:47146)
0619 08:58:31.225714 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:31.225725 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:31.225759 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:31.225894 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:31.225895 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:31.226091 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:58:31.226143 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:31.226145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.227071 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.227072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:31.227968 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:31.227980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:31.228468 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:31.228469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:31.228530 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:31.229024 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:58:31.229029 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:58:31.230578 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:58:31.232336 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:31.232337 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:31.232337 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:31.232555 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:31.232756 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:31.232757 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:31.232758 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:31.232819 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:31.233092 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:58:31.233093 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:58:31.233140 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 08:58:31.233206 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 08:58:31.233299 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 08:58:31.235507 13852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96ea9a7b62c04a32b67707dc093f4d50: Bootstrap starting.
I20260619 08:58:31.236290 13852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96ea9a7b62c04a32b67707dc093f4d50: Neither blocks nor log segments found. Creating new log.
I20260619 08:58:31.236585 13852 log.cc:824] T 00000000000000000000000000000000 P 96ea9a7b62c04a32b67707dc093f4d50: Log is configured to *not* fsync() on all Append() calls
I20260619 08:58:31.237394 13852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96ea9a7b62c04a32b67707dc093f4d50: No bootstrap required, opened a new log
I20260619 08:58:31.239315 13852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96ea9a7b62c04a32b67707dc093f4d50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8647fa46d6bf408ca30acfc775dc14aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41585 } } peers { permanent_uuid: "96ea9a7b62c04a32b67707dc093f4d50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38621 } } peers { permanent_uuid: "125f45ea6d3b4b338ded928eb53369f9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43533 } }
I20260619 08:58:31.239452 13852 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96ea9a7b62c04a32b67707dc093f4d50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:58:31.239485 13852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96ea9a7b62c04a32b67707dc093f4d50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96ea9a7b62c04a32b67707dc093f4d50, State: Initialized, Role: FOLLOWER
I20260619 08:58:31.239622 13852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96ea9a7b62c04a32b67707dc093f4d50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8647fa46d6bf408ca30acfc775dc14aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41585 } } peers { permanent_uuid: "96ea9a7b62c04a32b67707dc093f4d50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38621 } } peers { permanent_uuid: "125f45ea6d3b4b338ded928eb53369f9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43533 } }
I20260619 08:58:31.239914 13852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96ea9a7b62c04a32b67707dc093f4d50 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:58:31.239883 13969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96ea9a7b62c04a32b67707dc093f4d50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8647fa46d6bf408ca30acfc775dc14aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41585 } } peers { permanent_uuid: "96ea9a7b62c04a32b67707dc093f4d50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38621 } } peers { permanent_uuid: "125f45ea6d3b4b338ded928eb53369f9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43533 } } }
I20260619 08:58:31.240062 13969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96ea9a7b62c04a32b67707dc093f4d50 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 08:58:31.241850 13980 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 96ea9a7b62c04a32b67707dc093f4d50: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 08:58:31.241961 13980 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 08:58:31.243840 13784 master_runner.cc:263] No leader master found from master 96ea9a7b62c04a32b67707dc093f4d50
I20260619 08:58:31.243897 13784 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 08:58:31.347599 13730 tablet.cc:2369] T 00000000000000000000000000000000 P 8647fa46d6bf408ca30acfc775dc14aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 08:58:31.386214 13880 tablet.cc:2369] T 00000000000000000000000000000000 P 125f45ea6d3b4b338ded928eb53369f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:58:31.394840 13934 raft_consensus.cc:482] T 00000000000000000000000000000000 P 125f45ea6d3b4b338ded928eb53369f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:58:31.394994 13934 raft_consensus.cc:504] T 00000000000000000000000000000000 P 125f45ea6d3b4b338ded928eb53369f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8647fa46d6bf408ca30acfc775dc14aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41585 } } peers { permanent_uuid: "96ea9a7b62c04a32b67707dc093f4d50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38621 } } peers { permanent_uuid: "125f45ea6d3b4b338ded928eb53369f9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43533 } }
I20260619 08:58:31.395218 13934 leader_election.cc:288] T 00000000000000000000000000000000 P 125f45ea6d3b4b338ded928eb53369f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8647fa46d6bf408ca30acfc775dc14aa (127.0.98.62:41585), 96ea9a7b62c04a32b67707dc093f4d50 (127.0.98.61:38621)
I20260619 08:58:31.395510 13826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "125f45ea6d3b4b338ded928eb53369f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96ea9a7b62c04a32b67707dc093f4d50" is_pre_election: true
I20260619 08:58:31.395502 13751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "125f45ea6d3b4b338ded928eb53369f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8647fa46d6bf408ca30acfc775dc14aa" is_pre_election: true
I20260619 08:58:31.395641 13826 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 96ea9a7b62c04a32b67707dc093f4d50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125f45ea6d3b4b338ded928eb53369f9 in term 0.
I20260619 08:58:31.395664 13751 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8647fa46d6bf408ca30acfc775dc14aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 125f45ea6d3b4b338ded928eb53369f9 in term 0.
I20260619 08:58:31.395869 13878 leader_election.cc:302] T 00000000000000000000000000000000 P 125f45ea6d3b4b338ded928eb53369f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125f45ea6d3b4b338ded928eb53369f9, 96ea9a7b62c04a32b67707dc093f4d50; no voters:
I20260619 08:58:31.396001 13934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 125f45ea6d3b4b338ded928eb53369f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:58:31.396054 13934 raft_consensus.cc:482] T 00000000000000000000000000000000 P 125f45ea6d3b4b338ded928eb53369f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:58:31.396104 13934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 125f45ea6d3b4b338ded928eb53369f9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:58:31.396780 13934 raft_consensus.cc:504] T 00000000000000000000000000000000 P 125f45ea6d3b4b338ded928eb53369f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8647fa46d6bf408ca30acfc775dc14aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41585 } } peers { permanent_uuid: "96ea9a7b62c04a32b67707dc093f4d50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38621 } } peers { permanent_uuid: "125f45ea6d3b4b338ded928eb53369f9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43533 } }
I20260619 08:58:31.396889 13934 leader_election.cc:288] T 00000000000000000000000000000000 P 125f45ea6d3b4b338ded928eb53369f9 [CANDIDATE]: Term 1 election: Requested vote from peers 8647fa46d6bf408ca30acfc775dc14aa (127.0.98.62:41585), 96ea9a7b62c04a32b67707dc093f4d50 (127.0.98.61:38621)
I20260619 08:58:31.397099 13751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "125f45ea6d3b4b338ded928eb53369f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8647fa46d6bf408ca30acfc775dc14aa"
I20260619 08:58:31.397150 13826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "125f45ea6d3b4b338ded928eb53369f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96ea9a7b62c04a32b67707dc093f4d50"
I20260619 08:58:31.397185 13751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8647fa46d6bf408ca30acfc775dc14aa [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:58:31.397240 13826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96ea9a7b62c04a32b67707dc093f4d50 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:58:31.397888 13751 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8647fa46d6bf408ca30acfc775dc14aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125f45ea6d3b4b338ded928eb53369f9 in term 1.
I20260619 08:58:31.398048 13878 leader_election.cc:302] T 00000000000000000000000000000000 P 125f45ea6d3b4b338ded928eb53369f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 125f45ea6d3b4b338ded928eb53369f9, 8647fa46d6bf408ca30acfc775dc14aa; no voters:
I20260619 08:58:31.398150 13934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 125f45ea6d3b4b338ded928eb53369f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:58:31.398293 13934 raft_consensus.cc:686] T 00000000000000000000000000000000 P 125f45ea6d3b4b338ded928eb53369f9 [term 1 LEADER]: Becoming Leader. State: Replica: 125f45ea6d3b4b338ded928eb53369f9, State: Running, Role: LEADER
I20260619 08:58:31.398402 13934 consensus_queue.cc:237] T 00000000000000000000000000000000 P 125f45ea6d3b4b338ded928eb53369f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8647fa46d6bf408ca30acfc775dc14aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41585 } } peers { permanent_uuid: "96ea9a7b62c04a32b67707dc093f4d50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38621 } } peers { permanent_uuid: "125f45ea6d3b4b338ded928eb53369f9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43533 } }
I20260619 08:58:31.398485 13826 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 96ea9a7b62c04a32b67707dc093f4d50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 125f45ea6d3b4b338ded928eb53369f9 in term 1.
I20260619 08:58:31.398830 13982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 125f45ea6d3b4b338ded928eb53369f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 125f45ea6d3b4b338ded928eb53369f9. Latest consensus state: current_term: 1 leader_uuid: "125f45ea6d3b4b338ded928eb53369f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8647fa46d6bf408ca30acfc775dc14aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41585 } } peers { permanent_uuid: "96ea9a7b62c04a32b67707dc093f4d50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38621 } } peers { permanent_uuid: "125f45ea6d3b4b338ded928eb53369f9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43533 } } }
I20260619 08:58:31.398927 13982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 125f45ea6d3b4b338ded928eb53369f9 [sys.catalog]: This master's current role is: LEADER
I20260619 08:58:31.399132 13984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:58:31.399312 13984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:58:31.400593 13826 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 96ea9a7b62c04a32b67707dc093f4d50 [term 1 FOLLOWER]: Refusing update from remote peer 125f45ea6d3b4b338ded928eb53369f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:58:31.400593 13751 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8647fa46d6bf408ca30acfc775dc14aa [term 1 FOLLOWER]: Refusing update from remote peer 125f45ea6d3b4b338ded928eb53369f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:58:31.400880 13934 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 125f45ea6d3b4b338ded928eb53369f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8647fa46d6bf408ca30acfc775dc14aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:58:31.400990 13982 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 125f45ea6d3b4b338ded928eb53369f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96ea9a7b62c04a32b67707dc093f4d50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:58:31.402078 13969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96ea9a7b62c04a32b67707dc093f4d50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 125f45ea6d3b4b338ded928eb53369f9. Latest consensus state: current_term: 1 leader_uuid: "125f45ea6d3b4b338ded928eb53369f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8647fa46d6bf408ca30acfc775dc14aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41585 } } peers { permanent_uuid: "96ea9a7b62c04a32b67707dc093f4d50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38621 } } peers { permanent_uuid: "125f45ea6d3b4b338ded928eb53369f9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43533 } } }
I20260619 08:58:31.402160 13969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96ea9a7b62c04a32b67707dc093f4d50 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 08:58:31.402107 13955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8647fa46d6bf408ca30acfc775dc14aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 125f45ea6d3b4b338ded928eb53369f9. Latest consensus state: current_term: 1 leader_uuid: "125f45ea6d3b4b338ded928eb53369f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8647fa46d6bf408ca30acfc775dc14aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41585 } } peers { permanent_uuid: "96ea9a7b62c04a32b67707dc093f4d50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38621 } } peers { permanent_uuid: "125f45ea6d3b4b338ded928eb53369f9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43533 } } }
I20260619 08:58:31.402184 13955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8647fa46d6bf408ca30acfc775dc14aa [sys.catalog]: This master's current role is: FOLLOWER
I20260619 08:58:31.402879 13934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 125f45ea6d3b4b338ded928eb53369f9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "125f45ea6d3b4b338ded928eb53369f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8647fa46d6bf408ca30acfc775dc14aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41585 } } peers { permanent_uuid: "96ea9a7b62c04a32b67707dc093f4d50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38621 } } peers { permanent_uuid: "125f45ea6d3b4b338ded928eb53369f9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43533 } } }
I20260619 08:58:31.402951 13934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 125f45ea6d3b4b338ded928eb53369f9 [sys.catalog]: This master's current role is: LEADER
I20260619 08:58:31.403051 13985 mvcc.cc:205] Tried to move back new op lower bound from 7298496558694600704 to 7298496558688251904. Current Snapshot: MvccSnapshot[applied={T|T < 7298496558694600704}]
I20260619 08:58:31.403234 13934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 125f45ea6d3b4b338ded928eb53369f9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "125f45ea6d3b4b338ded928eb53369f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8647fa46d6bf408ca30acfc775dc14aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41585 } } peers { permanent_uuid: "96ea9a7b62c04a32b67707dc093f4d50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38621 } } peers { permanent_uuid: "125f45ea6d3b4b338ded928eb53369f9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43533 } } }
I20260619 08:58:31.403301 13934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 125f45ea6d3b4b338ded928eb53369f9 [sys.catalog]: This master's current role is: LEADER
I20260619 08:58:31.403232 13984 catalog_manager.cc:1347] Generated new cluster ID: 25a5dd47c5bb4023aabac4f8604f6fd3
I20260619 08:58:31.403589 13969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96ea9a7b62c04a32b67707dc093f4d50 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "125f45ea6d3b4b338ded928eb53369f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8647fa46d6bf408ca30acfc775dc14aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41585 } } peers { permanent_uuid: "96ea9a7b62c04a32b67707dc093f4d50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38621 } } peers { permanent_uuid: "125f45ea6d3b4b338ded928eb53369f9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43533 } } }
I20260619 08:58:31.403645 13984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:58:31.403656 13969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96ea9a7b62c04a32b67707dc093f4d50 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 08:58:31.404376 13955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8647fa46d6bf408ca30acfc775dc14aa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "125f45ea6d3b4b338ded928eb53369f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8647fa46d6bf408ca30acfc775dc14aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41585 } } peers { permanent_uuid: "96ea9a7b62c04a32b67707dc093f4d50" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38621 } } peers { permanent_uuid: "125f45ea6d3b4b338ded928eb53369f9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43533 } } }
I20260619 08:58:31.404462 13987 mvcc.cc:205] Tried to move back new op lower bound from 7298496558694600704 to 7298496558688251904. Current Snapshot: MvccSnapshot[applied={T|T < 7298496558694600704}]
I20260619 08:58:31.404475 13955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8647fa46d6bf408ca30acfc775dc14aa [sys.catalog]: This master's current role is: FOLLOWER
I20260619 08:58:31.418511 13984 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:58:31.419087 13984 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:58:31.421957 13984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 125f45ea6d3b4b338ded928eb53369f9: Generated new TSK 0
I20260619 08:58:31.422163 13984 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:58:32.206409 13966 catalog_manager.cc:1259] Loaded cluster ID: 25a5dd47c5bb4023aabac4f8604f6fd3
I20260619 08:58:32.206503 13966 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8647fa46d6bf408ca30acfc775dc14aa: loading cluster ID for follower catalog manager: success
I20260619 08:58:32.207363 13966 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8647fa46d6bf408ca30acfc775dc14aa: acquiring CA information for follower catalog manager: success
I20260619 08:58:32.207661 13966 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8647fa46d6bf408ca30acfc775dc14aa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 08:58:32.242444 13980 catalog_manager.cc:1259] Loaded cluster ID: 25a5dd47c5bb4023aabac4f8604f6fd3
I20260619 08:58:32.242527 13980 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 96ea9a7b62c04a32b67707dc093f4d50: loading cluster ID for follower catalog manager: success
I20260619 08:58:32.243299 13980 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 96ea9a7b62c04a32b67707dc093f4d50: acquiring CA information for follower catalog manager: success
I20260619 08:58:32.243556 13980 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 96ea9a7b62c04a32b67707dc093f4d50: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 08:58:32.475646 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 13709
I20260619 08:58:32.480427 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 13784
I20260619 08:58:32.484426 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 13859
2026-06-19T08:58:32Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2860 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:58:32 dist-test-slave-h4g5 krb5kdc[14000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:58:32 dist-test-slave-h4g5 krb5kdc[14000](info): set up 2 sockets
Jun 19 08:58:32 dist-test-slave-h4g5 krb5kdc[14000](info): commencing operation
krb5kdc: starting...
W20260619 08:58:33.439901 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 08:58:33 dist-test-slave-h4g5 krb5kdc[14000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:58:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:58:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 08:58:33.551713 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36259
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:58:33.630167 14013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:58:33.630378 14013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:58:33.630398 14013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:58:33.631546 14013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:58:33.631590 14013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:58:33.631603 14013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:58:33.631615 14013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:58:33.631625 14013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:58:33.633081 14013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:58:33.633277 14013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:58:33.633450 14013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:58:33.638067 14019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:33.638067 14018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:33.638150 14021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:33.638219 14013 server_base.cc:1034] running on GCE node
I20260619 08:58:33.638667 14013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:58:33.638916 14013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:58:33.640050 14013 hybrid_clock.cc:648] HybridClock initialized: now 1781859513640037 us; error 29 us; skew 500 ppm
Jun 19 08:58:33 dist-test-slave-h4g5 krb5kdc[14000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:58:33.642478 14013 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 08:58:33.642944 14013 webserver.cc:460] Webserver started at http://127.0.98.62:46447/ using document root <none> and password file <none>
I20260619 08:58:33.643157 14013 fs_manager.cc:359] Metadata directory not provided
I20260619 08:58:33.643203 14013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:58:33.643326 14013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:58:33.644416 14013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "6fe944016d4141678ecdb3e022798164"
format_stamp: "Formatted at 2026-06-19 08:58:33 on dist-test-slave-h4g5"
server_key: "da95bc8c8199c11d82b8bba3c833a019"
server_key_iv: "959ddb3510489f61591a6b7000945cfe"
server_key_version: "encryptionkey@0"
I20260619 08:58:33.644750 14013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "6fe944016d4141678ecdb3e022798164"
format_stamp: "Formatted at 2026-06-19 08:58:33 on dist-test-slave-h4g5"
server_key: "da95bc8c8199c11d82b8bba3c833a019"
server_key_iv: "959ddb3510489f61591a6b7000945cfe"
server_key_version: "encryptionkey@0"
I20260619 08:58:33.645956 14013 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:58:33.646615 14028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:33.646773 14013 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:58:33.646857 14013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "6fe944016d4141678ecdb3e022798164"
format_stamp: "Formatted at 2026-06-19 08:58:33 on dist-test-slave-h4g5"
server_key: "da95bc8c8199c11d82b8bba3c833a019"
server_key_iv: "959ddb3510489f61591a6b7000945cfe"
server_key_version: "encryptionkey@0"
I20260619 08:58:33.646916 14013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:58:33.656559 14013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:58:33.658666 14013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:58:33.658802 14013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:58:33.662472 14013 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36259
I20260619 08:58:33.662516 14080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36259 every 8 connection(s)
I20260619 08:58:33.662838 14013 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:58:33.663631 14081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:58:33.665879 14081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fe944016d4141678ecdb3e022798164: Bootstrap starting.
I20260619 08:58:33.666144 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 14013
I20260619 08:58:33.666226 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 08:58:33.666402 392 external_mini_cluster.cc:1428] Setting key f0bf96a6abb3eb37a8929189e2198a33
I20260619 08:58:33.666513 14081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fe944016d4141678ecdb3e022798164: Neither blocks nor log segments found. Creating new log.
I20260619 08:58:33.666831 14081 log.cc:824] T 00000000000000000000000000000000 P 6fe944016d4141678ecdb3e022798164: Log is configured to *not* fsync() on all Append() calls
I20260619 08:58:33.667639 14081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fe944016d4141678ecdb3e022798164: No bootstrap required, opened a new log
I20260619 08:58:33.669919 14081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fe944016d4141678ecdb3e022798164 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fe944016d4141678ecdb3e022798164" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36259 } }
I20260619 08:58:33.670074 14081 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fe944016d4141678ecdb3e022798164 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:58:33.670109 14081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fe944016d4141678ecdb3e022798164 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fe944016d4141678ecdb3e022798164, State: Initialized, Role: FOLLOWER
I20260619 08:58:33.670233 14081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fe944016d4141678ecdb3e022798164 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fe944016d4141678ecdb3e022798164" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36259 } }
I20260619 08:58:33.670305 14081 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fe944016d4141678ecdb3e022798164 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:58:33.670336 14081 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fe944016d4141678ecdb3e022798164 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:58:33.670382 14081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fe944016d4141678ecdb3e022798164 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:58:33.670958 14081 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fe944016d4141678ecdb3e022798164 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fe944016d4141678ecdb3e022798164" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36259 } }
I20260619 08:58:33.671065 14081 leader_election.cc:302] T 00000000000000000000000000000000 P 6fe944016d4141678ecdb3e022798164 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fe944016d4141678ecdb3e022798164; no voters:
I20260619 08:58:33.671224 14081 leader_election.cc:288] T 00000000000000000000000000000000 P 6fe944016d4141678ecdb3e022798164 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:58:33.671276 14086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fe944016d4141678ecdb3e022798164 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:58:33.671443 14086 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fe944016d4141678ecdb3e022798164 [term 1 LEADER]: Becoming Leader. State: Replica: 6fe944016d4141678ecdb3e022798164, State: Running, Role: LEADER
I20260619 08:58:33.671480 14081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fe944016d4141678ecdb3e022798164 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:58:33.671574 14086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fe944016d4141678ecdb3e022798164 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fe944016d4141678ecdb3e022798164" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36259 } }
I20260619 08:58:33.671947 14086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fe944016d4141678ecdb3e022798164 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fe944016d4141678ecdb3e022798164. Latest consensus state: current_term: 1 leader_uuid: "6fe944016d4141678ecdb3e022798164" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fe944016d4141678ecdb3e022798164" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36259 } } }
I20260619 08:58:33.671981 14087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fe944016d4141678ecdb3e022798164 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fe944016d4141678ecdb3e022798164" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fe944016d4141678ecdb3e022798164" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36259 } } }
I20260619 08:58:33.672173 14087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fe944016d4141678ecdb3e022798164 [sys.catalog]: This master's current role is: LEADER
I20260619 08:58:33.672360 14086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fe944016d4141678ecdb3e022798164 [sys.catalog]: This master's current role is: LEADER
I20260619 08:58:33.672580 14095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:58:33.673297 14095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 08:58:33 dist-test-slave-h4g5 krb5kdc[14000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:58:33.674911 14095 catalog_manager.cc:1347] Generated new cluster ID: 054087e29a7749308ec5ff4d06c99301
I20260619 08:58:33.674960 14095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:58:33.678077 14084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:33.666862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47238 (local address 127.0.98.62:36259)
0619 08:58:33.667039 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:33.667051 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:33.667088 (+ 37us) server_negotiation.cc:407] Connection header received
0619 08:58:33.667584 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:33.667593 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:33.667652 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:33.667820 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:33.668463 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:33.669858 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:33.670731 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:33.671346 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:33.671455 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:33.674304 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:33.674308 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:33.674308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:33.674321 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:33.676431 (+ 2110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:33.676828 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:33.676831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:33.676831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:33.676944 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:33.677120 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:33.677120 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:33.677121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:33.677457 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:33.677641 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:33.677724 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 08:58:33.677863 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
I20260619 08:58:33.679039 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 08:58:33.688817 14095 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:58:33.689323 14095 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:58:33.701148 14095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fe944016d4141678ecdb3e022798164: Generated new TSK 0
I20260619 08:58:33.701393 14095 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:58:33.732923 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 14013
2026-06-19T08:58:33Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1247 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:58:33 dist-test-slave-h4g5 krb5kdc[14108](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:58:33 dist-test-slave-h4g5 krb5kdc[14108](info): set up 2 sockets
Jun 19 08:58:33 dist-test-slave-h4g5 krb5kdc[14108](info): commencing operation
krb5kdc: starting...
W20260619 08:58:34.679863 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 08:58:34 dist-test-slave-h4g5 krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:58:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:58:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 08:58:34.791285 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46369
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46369
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:58:34.868396 14121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:58:34.868633 14121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:58:34.868664 14121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:58:34.869799 14121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:58:34.869838 14121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:58:34.869858 14121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:58:34.869874 14121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:58:34.869890 14121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:58:34.871366 14121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:58:34.871608 14121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:58:34.871798 14121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:58:34.876367 14129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:34.876497 14121 server_base.cc:1034] running on GCE node
W20260619 08:58:34.876389 14126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:34.876400 14127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:34.876765 14121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:58:34.877012 14121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:58:34.878165 14121 hybrid_clock.cc:648] HybridClock initialized: now 1781859514878150 us; error 28 us; skew 500 ppm
Jun 19 08:58:34 dist-test-slave-h4g5 krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:58:34.880596 14121 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 08:58:34.881021 14121 webserver.cc:460] Webserver started at http://127.0.98.62:34429/ using document root <none> and password file <none>
I20260619 08:58:34.881225 14121 fs_manager.cc:359] Metadata directory not provided
I20260619 08:58:34.881274 14121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:58:34.881405 14121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:58:34.882413 14121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "8991bbb9aea44da8a3e5b238d53a7593"
format_stamp: "Formatted at 2026-06-19 08:58:34 on dist-test-slave-h4g5"
server_key: "5a56c1513c720f332d576105d2e023d3"
server_key_iv: "3de4fa3ad67e8e0a96dab081147f7383"
server_key_version: "encryptionkey@0"
I20260619 08:58:34.882730 14121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "8991bbb9aea44da8a3e5b238d53a7593"
format_stamp: "Formatted at 2026-06-19 08:58:34 on dist-test-slave-h4g5"
server_key: "5a56c1513c720f332d576105d2e023d3"
server_key_iv: "3de4fa3ad67e8e0a96dab081147f7383"
server_key_version: "encryptionkey@0"
I20260619 08:58:34.883948 14121 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 08:58:34.884703 14136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:34.884848 14121 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:34.884927 14121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "8991bbb9aea44da8a3e5b238d53a7593"
format_stamp: "Formatted at 2026-06-19 08:58:34 on dist-test-slave-h4g5"
server_key: "5a56c1513c720f332d576105d2e023d3"
server_key_iv: "3de4fa3ad67e8e0a96dab081147f7383"
server_key_version: "encryptionkey@0"
I20260619 08:58:34.884991 14121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:58:34.891384 14121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:58:34.892980 14121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:58:34.893098 14121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:58:34.896605 14121 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46369
I20260619 08:58:34.896631 14188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46369 every 8 connection(s)
I20260619 08:58:34.896922 14121 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:58:34.897822 14189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:58:34.899911 14189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593: Bootstrap starting.
I20260619 08:58:34.900467 14189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593: Neither blocks nor log segments found. Creating new log.
I20260619 08:58:34.900702 14189 log.cc:824] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593: Log is configured to *not* fsync() on all Append() calls
I20260619 08:58:34.901304 14189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593: No bootstrap required, opened a new log
I20260619 08:58:34.902817 14189 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8991bbb9aea44da8a3e5b238d53a7593" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46369 } }
I20260619 08:58:34.902951 14189 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:58:34.902971 14189 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8991bbb9aea44da8a3e5b238d53a7593, State: Initialized, Role: FOLLOWER
I20260619 08:58:34.903093 14189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8991bbb9aea44da8a3e5b238d53a7593" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46369 } }
I20260619 08:58:34.903151 14189 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:58:34.903172 14189 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:58:34.903216 14189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:58:34.903971 14189 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8991bbb9aea44da8a3e5b238d53a7593" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46369 } }
I20260619 08:58:34.904059 14189 leader_election.cc:302] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8991bbb9aea44da8a3e5b238d53a7593; no voters:
I20260619 08:58:34.904250 14189 leader_election.cc:288] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:58:34.904287 14192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:58:34.904425 14192 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [term 1 LEADER]: Becoming Leader. State: Replica: 8991bbb9aea44da8a3e5b238d53a7593, State: Running, Role: LEADER
I20260619 08:58:34.904487 14189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:58:34.904510 14192 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8991bbb9aea44da8a3e5b238d53a7593" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46369 } }
I20260619 08:58:34.904832 14193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8991bbb9aea44da8a3e5b238d53a7593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8991bbb9aea44da8a3e5b238d53a7593" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46369 } } }
I20260619 08:58:34.904842 14194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8991bbb9aea44da8a3e5b238d53a7593. Latest consensus state: current_term: 1 leader_uuid: "8991bbb9aea44da8a3e5b238d53a7593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8991bbb9aea44da8a3e5b238d53a7593" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46369 } } }
I20260619 08:58:34.904953 14193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [sys.catalog]: This master's current role is: LEADER
I20260619 08:58:34.905205 14200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:58:34.905325 14194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [sys.catalog]: This master's current role is: LEADER
I20260619 08:58:34.905519 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 14121
I20260619 08:58:34.905602 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 08:58:34.905705 14200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:58:34.905797 392 external_mini_cluster.cc:1428] Setting key 707ceb7b16582519077d4b2ff8ca09f9
I20260619 08:58:34.907145 14200 catalog_manager.cc:1347] Generated new cluster ID: e8c4656838104c6894f1c2d9db3427b1
I20260619 08:58:34.907192 14200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 08:58:34 dist-test-slave-h4g5 krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:58:34.915105 14200 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:58:34.915791 14200 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:58:34.917440 14211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:34.906226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47866 (local address 127.0.98.62:46369)
0619 08:58:34.906385 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:34.906395 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:34.906438 (+ 43us) server_negotiation.cc:407] Connection header received
0619 08:58:34.906539 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:34.906544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:34.906593 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:34.906710 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:34.907454 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:34.908827 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:34.909887 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:34.910467 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:34.910564 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:34.913677 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:34.913737 (+ 60us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:34.913738 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:34.913750 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:34.915768 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:34.916173 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:34.916176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:34.916176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:34.916272 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:34.916467 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:34.916467 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:34.916468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:34.916715 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:34.916824 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:34.917055 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 08:58:34.917206 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
I20260619 08:58:34.929605 14200 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593: Generated new TSK 0
I20260619 08:58:34.929782 14200 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 08:58:34.955367 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46369
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:58:35.033819 14215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:58:35.034021 14215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:58:35.034052 14215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:58:35.035184 14215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:58:35.035224 14215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:58:35.035259 14215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 08:58:35.036731 14215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:58:35.036960 14215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:58:35.037170 14215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:58:35.041841 14223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:35.041893 14215 server_base.cc:1034] running on GCE node
W20260619 08:58:35.041987 14221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:35.041967 14220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:35.042237 14215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:58:35.042439 14215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:58:35.043577 14215 hybrid_clock.cc:648] HybridClock initialized: now 1781859515043563 us; error 28 us; skew 500 ppm
Jun 19 08:58:35 dist-test-slave-h4g5 krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:58:35.045944 14215 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 08:58:35.046324 14215 webserver.cc:460] Webserver started at http://127.0.98.1:45279/ using document root <none> and password file <none>
I20260619 08:58:35.046522 14215 fs_manager.cc:359] Metadata directory not provided
I20260619 08:58:35.046568 14215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:58:35.046689 14215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:58:35.047657 14215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/data/instance:
uuid: "1d9e834a6cb74090b30ff7e5b108e954"
format_stamp: "Formatted at 2026-06-19 08:58:35 on dist-test-slave-h4g5"
server_key: "a2290833a50be30419b4b21f5df4a2f9"
server_key_iv: "5c02e885c7370220c31eb5030831c96d"
server_key_version: "encryptionkey@0"
I20260619 08:58:35.047964 14215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1d9e834a6cb74090b30ff7e5b108e954"
format_stamp: "Formatted at 2026-06-19 08:58:35 on dist-test-slave-h4g5"
server_key: "a2290833a50be30419b4b21f5df4a2f9"
server_key_iv: "5c02e885c7370220c31eb5030831c96d"
server_key_version: "encryptionkey@0"
I20260619 08:58:35.049141 14215 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:58:35.049798 14230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:35.049953 14215 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 08:58:35.050031 14215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/wal
uuid: "1d9e834a6cb74090b30ff7e5b108e954"
format_stamp: "Formatted at 2026-06-19 08:58:35 on dist-test-slave-h4g5"
server_key: "a2290833a50be30419b4b21f5df4a2f9"
server_key_iv: "5c02e885c7370220c31eb5030831c96d"
server_key_version: "encryptionkey@0"
I20260619 08:58:35.050096 14215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:58:35.059401 14215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:58:35.061127 14215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:58:35.061260 14215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:58:35.061463 14215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:58:35.061734 14215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:58:35.061766 14215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:35.061794 14215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:58:35.061825 14215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:35.066931 14215 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44613
I20260619 08:58:35.066999 14343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44613 every 8 connection(s)
I20260619 08:58:35.067252 14215 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
I20260619 08:58:35.070000 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 14215
I20260619 08:58:35.070092 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/wal/instance
I20260619 08:58:35.070261 392 external_mini_cluster.cc:1428] Setting key 880322198f21c92e339e983577de88d3
Jun 19 08:58:35 dist-test-slave-h4g5 krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:58:35.077054 14211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:35.067912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60017 (local address 127.0.98.62:46369)
0619 08:58:35.067972 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:35.067974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:35.068170 (+ 196us) server_negotiation.cc:407] Connection header received
0619 08:58:35.068690 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:35.068691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:35.068703 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:35.068755 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:35.069666 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.070211 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.071403 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.071607 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.071664 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:35.073628 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:35.073630 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:35.073630 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:35.073637 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:35.075795 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:35.076165 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:35.076166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:35.076166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:35.076223 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:35.076432 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:35.076433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:35.076433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:35.076543 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:35.076596 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:35.076858 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 08:58:35.076960 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 08:58:35.077167 14346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:35.067872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.1:60017)
0619 08:58:35.068030 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:35.068069 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:35.068589 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:35.068835 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:35.068839 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:35.069377 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:58:35.069569 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:35.069574 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.070301 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.070302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:35.071319 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:35.071322 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.071673 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.071674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:35.071805 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:35.072286 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:58:35.072294 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:58:35.073549 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:58:35.075868 (+ 2319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:35.075876 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:35.075878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:35.076107 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:35.076273 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:35.076274 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:35.076274 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:35.076370 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:35.076594 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:58:35.076600 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:58:35.076726 (+ 126us) client_negotiation.cc:769] Sending connection context
0619 08:58:35.076806 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 08:58:35.076959 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 08:58:35.077556 14344 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46369
I20260619 08:58:35.077641 14344 heartbeater.cc:464] Registering TS with master...
I20260619 08:58:35.077803 14344 heartbeater.cc:511] Master 127.0.98.62:46369 requested a full tablet report, sending...
I20260619 08:58:35.078263 14153 ts_manager.cc:194] Registered new tserver with Master: 1d9e834a6cb74090b30ff7e5b108e954 (127.0.98.1:44613)
I20260619 08:58:35.078842 14153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60017
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 08:58:35.094069 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46369
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:58:35.173444 14349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:58:35.173641 14349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:58:35.173661 14349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:58:35.174780 14349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:58:35.174818 14349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:58:35.174842 14349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 08:58:35.176244 14349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:58:35.176465 14349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:58:35.176672 14349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 08:58:35.181514 14349 server_base.cc:1034] running on GCE node
W20260619 08:58:35.181489 14357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:35.181478 14354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:35.181483 14355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:35.181994 14349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:58:35.182255 14349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:58:35.183414 14349 hybrid_clock.cc:648] HybridClock initialized: now 1781859515183400 us; error 27 us; skew 500 ppm
Jun 19 08:58:35 dist-test-slave-h4g5 krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:58:35.185849 14349 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 08:58:35.186231 14349 webserver.cc:460] Webserver started at http://127.0.98.2:34677/ using document root <none> and password file <none>
I20260619 08:58:35.186399 14349 fs_manager.cc:359] Metadata directory not provided
I20260619 08:58:35.186436 14349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:58:35.186527 14349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:58:35.187587 14349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/data/instance:
uuid: "b0bb3b26f9e54bc8b16d05572a939d7d"
format_stamp: "Formatted at 2026-06-19 08:58:35 on dist-test-slave-h4g5"
server_key: "caf54e9d0162b428c8b0e0f052afde4d"
server_key_iv: "e2af6d8a5ebb5c02ff8bc957fce5b335"
server_key_version: "encryptionkey@0"
I20260619 08:58:35.187907 14349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b0bb3b26f9e54bc8b16d05572a939d7d"
format_stamp: "Formatted at 2026-06-19 08:58:35 on dist-test-slave-h4g5"
server_key: "caf54e9d0162b428c8b0e0f052afde4d"
server_key_iv: "e2af6d8a5ebb5c02ff8bc957fce5b335"
server_key_version: "encryptionkey@0"
I20260619 08:58:35.189221 14349 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:58:35.189908 14364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:35.190106 14349 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:58:35.190179 14349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/wal
uuid: "b0bb3b26f9e54bc8b16d05572a939d7d"
format_stamp: "Formatted at 2026-06-19 08:58:35 on dist-test-slave-h4g5"
server_key: "caf54e9d0162b428c8b0e0f052afde4d"
server_key_iv: "e2af6d8a5ebb5c02ff8bc957fce5b335"
server_key_version: "encryptionkey@0"
I20260619 08:58:35.190294 14349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:58:35.213814 14349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:58:35.215909 14349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:58:35.216042 14349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:58:35.216286 14349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:58:35.216570 14349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:58:35.216601 14349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:35.216622 14349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:58:35.216645 14349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:35.221915 14349 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43053
I20260619 08:58:35.221978 14477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43053 every 8 connection(s)
I20260619 08:58:35.222258 14349 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 08:58:35 dist-test-slave-h4g5 krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:58:35.228649 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 14349
I20260619 08:58:35.228746 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/wal/instance
I20260619 08:58:35.228912 392 external_mini_cluster.cc:1428] Setting key e0df64b72b489e02e29acada7885f467
I20260619 08:58:35.231082 14211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:35.223039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45185 (local address 127.0.98.62:46369)
0619 08:58:35.223114 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:35.223116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:35.223314 (+ 198us) server_negotiation.cc:407] Connection header received
0619 08:58:35.223803 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:35.223804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:35.223815 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:35.223860 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:35.224823 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.225325 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.226271 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.226450 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.226498 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:35.228471 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:35.228473 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:35.228473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:35.228480 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:35.229876 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:35.230228 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:35.230229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:35.230230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:35.230283 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:35.230496 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:35.230497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:35.230497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:35.230604 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:35.230657 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:35.230915 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 08:58:35.230994 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 08:58:35.231274 14480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:35.223011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.2:45185)
0619 08:58:35.223182 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:35.223231 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:35.223723 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:35.223923 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:35.223926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:35.224528 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:58:35.224722 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:35.224727 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.225408 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.225409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:35.226202 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:35.226204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.226506 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.226507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:35.226600 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:35.227017 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:58:35.227025 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:58:35.228380 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:58:35.229949 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:35.229955 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:35.229957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:35.230163 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:35.230337 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:35.230338 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:35.230338 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:35.230441 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:35.230659 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:58:35.230665 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:58:35.230799 (+ 134us) client_negotiation.cc:769] Sending connection context
0619 08:58:35.230888 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 08:58:35.231039 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":85,"threads_started":1}
I20260619 08:58:35.232188 14478 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46369
I20260619 08:58:35.232288 14478 heartbeater.cc:464] Registering TS with master...
I20260619 08:58:35.232461 14478 heartbeater.cc:511] Master 127.0.98.62:46369 requested a full tablet report, sending...
I20260619 08:58:35.232822 14153 ts_manager.cc:194] Registered new tserver with Master: b0bb3b26f9e54bc8b16d05572a939d7d (127.0.98.2:43053)
I20260619 08:58:35.233174 14153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45185
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 08:58:35.253000 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46369
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:58:35.337041 14483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:58:35.337247 14483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:58:35.337270 14483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:58:35.338382 14483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:58:35.338421 14483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:58:35.338446 14483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 08:58:35.339871 14483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:58:35.340063 14483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:58:35.340293 14483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:58:35.345366 14488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:35.345476 14483 server_base.cc:1034] running on GCE node
W20260619 08:58:35.345340 14489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:35.345327 14491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:35.345732 14483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:58:35.346002 14483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:58:35.347170 14483 hybrid_clock.cc:648] HybridClock initialized: now 1781859515347156 us; error 29 us; skew 500 ppm
Jun 19 08:58:35 dist-test-slave-h4g5 krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:58:35.349812 14483 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 08:58:35.350275 14483 webserver.cc:460] Webserver started at http://127.0.98.3:32985/ using document root <none> and password file <none>
I20260619 08:58:35.350502 14483 fs_manager.cc:359] Metadata directory not provided
I20260619 08:58:35.350550 14483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:58:35.350677 14483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:58:35.351754 14483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/data/instance:
uuid: "ba1b76511e9a401fad69d6e7175118cb"
format_stamp: "Formatted at 2026-06-19 08:58:35 on dist-test-slave-h4g5"
server_key: "24e860de45ea5d1e84425175fae14ea6"
server_key_iv: "03c19c7894473b74e1e493c956ba66cb"
server_key_version: "encryptionkey@0"
I20260619 08:58:35.352169 14483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ba1b76511e9a401fad69d6e7175118cb"
format_stamp: "Formatted at 2026-06-19 08:58:35 on dist-test-slave-h4g5"
server_key: "24e860de45ea5d1e84425175fae14ea6"
server_key_iv: "03c19c7894473b74e1e493c956ba66cb"
server_key_version: "encryptionkey@0"
I20260619 08:58:35.353528 14483 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:58:35.354334 14498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:35.354501 14483 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:58:35.354578 14483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/wal
uuid: "ba1b76511e9a401fad69d6e7175118cb"
format_stamp: "Formatted at 2026-06-19 08:58:35 on dist-test-slave-h4g5"
server_key: "24e860de45ea5d1e84425175fae14ea6"
server_key_iv: "03c19c7894473b74e1e493c956ba66cb"
server_key_version: "encryptionkey@0"
I20260619 08:58:35.354642 14483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:58:35.371121 14483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:58:35.373160 14483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:58:35.373296 14483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:58:35.373512 14483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:58:35.373812 14483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:58:35.373845 14483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:35.373874 14483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:58:35.373890 14483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:35.379189 14483 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33255
I20260619 08:58:35.379230 14611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33255 every 8 connection(s)
I20260619 08:58:35.379534 14483 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
I20260619 08:58:35.381095 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 14483
I20260619 08:58:35.381204 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/wal/instance
I20260619 08:58:35.381438 392 external_mini_cluster.cc:1428] Setting key 0ec24af46fc07734ae687b5fd0cb648c
Jun 19 08:58:35 dist-test-slave-h4g5 krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:58:35.388801 14211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:35.380237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49597 (local address 127.0.98.62:46369)
0619 08:58:35.380318 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:35.380320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:35.380521 (+ 201us) server_negotiation.cc:407] Connection header received
0619 08:58:35.381064 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:35.381065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:35.381075 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:35.381115 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:35.382307 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.382819 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.383756 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.383927 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.383981 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:35.386302 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:35.386304 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:35.386304 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:35.386311 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:35.387571 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:35.387932 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:35.387932 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:35.387933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:35.387991 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:35.388264 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:35.388264 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:35.388265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:35.388365 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:35.388415 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:35.388679 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 08:58:35.388738 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 08:58:35.388996 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:35.380240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.3:49597)
0619 08:58:35.380401 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:35.380438 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:35.380954 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:35.381200 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:35.381204 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:35.381948 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:58:35.382219 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:35.382226 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.382894 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.382894 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:35.383694 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:35.383696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.384018 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.384019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:35.384191 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:35.384737 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:58:35.384757 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:58:35.386191 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:58:35.387650 (+ 1459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:35.387655 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:35.387657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:35.387871 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:35.388058 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:35.388059 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:35.388059 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:35.388206 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:35.388431 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:58:35.388435 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:58:35.388568 (+ 133us) client_negotiation.cc:769] Sending connection context
0619 08:58:35.388654 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 08:58:35.388774 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
I20260619 08:58:35.389675 14612 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46369
I20260619 08:58:35.389786 14612 heartbeater.cc:464] Registering TS with master...
I20260619 08:58:35.389963 14612 heartbeater.cc:511] Master 127.0.98.62:46369 requested a full tablet report, sending...
I20260619 08:58:35.390380 14153 ts_manager.cc:194] Registered new tserver with Master: ba1b76511e9a401fad69d6e7175118cb (127.0.98.3:33255)
I20260619 08:58:35.390794 14153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49597
I20260619 08:58:35.393429 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:58:35.400374 14211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:35.394754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47878 (local address 127.0.98.62:46369)
0619 08:58:35.394835 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:35.394836 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:35.394880 (+ 44us) server_negotiation.cc:407] Connection header received
0619 08:58:35.394929 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:35.394929 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:35.394937 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:35.394983 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:35.395572 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.396050 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.396819 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.397017 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.397061 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:35.398121 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:35.398122 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:35.398122 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:35.398127 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:35.399195 (+ 1068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:35.399521 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:35.399522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:35.399522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:35.399579 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:35.399811 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:35.399812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:35.399813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:35.399963 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:35.400051 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:35.400219 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 08:58:35.400285 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 08:58:35.401950 14153 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47878:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:58:35.403653 14153 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:58:35.410009 14621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:35.406064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33255 (local address 127.0.0.1:33846)
0619 08:58:35.406506 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:35.406515 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:35.406563 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:35.406847 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:35.406851 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:35.406853 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:58:35.406922 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:35.406925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.408309 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.408310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:35.409034 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:35.409036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.409696 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.409697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:35.409789 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:35.409803 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 08:58:35.409878 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 08:58:35.409933 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":57,"threads_started":1}
I20260619 08:58:35.410018 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:35.406174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33846 (local address 127.0.98.3:33255)
0619 08:58:35.406383 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:35.406387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:35.406609 (+ 222us) server_negotiation.cc:407] Connection header received
0619 08:58:35.406655 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:35.406658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:35.406676 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:35.406752 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:58:35.407005 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.408178 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.409122 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.409615 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.409687 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:35.409759 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:35.409892 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 08:58:35.409933 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":92,"threads_started":1}
I20260619 08:58:35.410351 14624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:35.406439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52662 (local address 127.0.98.1:44613)
0619 08:58:35.406789 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:35.406793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:35.406802 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:58:35.406841 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:35.406844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:35.406858 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:35.406930 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:58:35.407122 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.407796 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.409164 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.409909 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.409973 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:35.410026 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:35.410219 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 08:58:35.410268 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":73,"threads_started":1}
I20260619 08:58:35.410301 14623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:35.406317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44613 (local address 127.0.0.1:52662)
0619 08:58:35.406590 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:35.406600 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:35.406652 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:35.407013 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:35.407014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:35.407016 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:58:35.407054 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:35.407056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.407884 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.407885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:35.409075 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:35.409078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.410058 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.410059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:35.410125 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:35.410129 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:58:35.410180 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 08:58:35.410220 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":54,"threads_started":1}
I20260619 08:58:35.410709 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:35.406062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43053 (local address 127.0.0.1:40930)
0619 08:58:35.406366 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:35.406406 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:35.406472 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:35.406893 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:35.406894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:35.406895 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:58:35.406941 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:35.406942 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.408278 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.408279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:35.409630 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:35.409633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.410517 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.410517 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:35.410579 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:35.410583 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:58:35.410608 (+ 25us) client_negotiation.cc:240] Negotiation successful
0619 08:58:35.410644 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":31,"thread_start_us":59,"threads_started":1}
I20260619 08:58:35.411125 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:35.406535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40930 (local address 127.0.98.2:43053)
0619 08:58:35.406712 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:35.406716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:35.406726 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:58:35.406774 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:35.406776 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:35.406789 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:35.407014 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:58:35.407088 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.408190 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.409718 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.410421 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.410821 (+ 400us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:35.410988 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:35.411012 (+ 24us) server_negotiation.cc:299] Negotiation successful
0619 08:58:35.411050 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":87,"threads_started":1}
I20260619 08:58:35.411072 14278 tablet_service.cc:1467] Processing CreateTablet for tablet 7a1b355e49b342f28957f5e04ae95b18 (DEFAULT_TABLE table=test-table [id=635e7250c9c844bcb82097e30da5c2e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:58:35.411288 14546 tablet_service.cc:1467] Processing CreateTablet for tablet 7a1b355e49b342f28957f5e04ae95b18 (DEFAULT_TABLE table=test-table [id=635e7250c9c844bcb82097e30da5c2e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:58:35.411460 14278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a1b355e49b342f28957f5e04ae95b18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:58:35.411571 14546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a1b355e49b342f28957f5e04ae95b18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:58:35.411739 14412 tablet_service.cc:1467] Processing CreateTablet for tablet 7a1b355e49b342f28957f5e04ae95b18 (DEFAULT_TABLE table=test-table [id=635e7250c9c844bcb82097e30da5c2e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:58:35.412021 14412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a1b355e49b342f28957f5e04ae95b18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:58:35.414127 14626 tablet_bootstrap.cc:492] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954: Bootstrap starting.
I20260619 08:58:35.414770 14626 tablet_bootstrap.cc:654] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954: Neither blocks nor log segments found. Creating new log.
I20260619 08:58:35.414784 14627 tablet_bootstrap.cc:492] T 7a1b355e49b342f28957f5e04ae95b18 P ba1b76511e9a401fad69d6e7175118cb: Bootstrap starting.
I20260619 08:58:35.414971 14628 tablet_bootstrap.cc:492] T 7a1b355e49b342f28957f5e04ae95b18 P b0bb3b26f9e54bc8b16d05572a939d7d: Bootstrap starting.
I20260619 08:58:35.415452 14627 tablet_bootstrap.cc:654] T 7a1b355e49b342f28957f5e04ae95b18 P ba1b76511e9a401fad69d6e7175118cb: Neither blocks nor log segments found. Creating new log.
I20260619 08:58:35.415501 14628 tablet_bootstrap.cc:654] T 7a1b355e49b342f28957f5e04ae95b18 P b0bb3b26f9e54bc8b16d05572a939d7d: Neither blocks nor log segments found. Creating new log.
I20260619 08:58:35.415565 14626 log.cc:824] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954: Log is configured to *not* fsync() on all Append() calls
I20260619 08:58:35.415977 14627 log.cc:824] T 7a1b355e49b342f28957f5e04ae95b18 P ba1b76511e9a401fad69d6e7175118cb: Log is configured to *not* fsync() on all Append() calls
I20260619 08:58:35.415972 14628 log.cc:824] T 7a1b355e49b342f28957f5e04ae95b18 P b0bb3b26f9e54bc8b16d05572a939d7d: Log is configured to *not* fsync() on all Append() calls
I20260619 08:58:35.416458 14626 tablet_bootstrap.cc:492] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954: No bootstrap required, opened a new log
I20260619 08:58:35.416518 14626 ts_tablet_manager.cc:1397] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:58:35.416613 14628 tablet_bootstrap.cc:492] T 7a1b355e49b342f28957f5e04ae95b18 P b0bb3b26f9e54bc8b16d05572a939d7d: No bootstrap required, opened a new log
I20260619 08:58:35.416671 14628 ts_tablet_manager.cc:1397] T 7a1b355e49b342f28957f5e04ae95b18 P b0bb3b26f9e54bc8b16d05572a939d7d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:58:35.418097 14626 raft_consensus.cc:348] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43053 } } peers { permanent_uuid: "1d9e834a6cb74090b30ff7e5b108e954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44613 } } peers { permanent_uuid: "ba1b76511e9a401fad69d6e7175118cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } }
I20260619 08:58:35.418246 14626 raft_consensus.cc:374] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:58:35.418272 14626 raft_consensus.cc:729] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d9e834a6cb74090b30ff7e5b108e954, State: Initialized, Role: FOLLOWER
I20260619 08:58:35.418202 14628 raft_consensus.cc:348] T 7a1b355e49b342f28957f5e04ae95b18 P b0bb3b26f9e54bc8b16d05572a939d7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43053 } } peers { permanent_uuid: "1d9e834a6cb74090b30ff7e5b108e954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44613 } } peers { permanent_uuid: "ba1b76511e9a401fad69d6e7175118cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } }
I20260619 08:58:35.418370 14628 raft_consensus.cc:374] T 7a1b355e49b342f28957f5e04ae95b18 P b0bb3b26f9e54bc8b16d05572a939d7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:58:35.418408 14628 raft_consensus.cc:729] T 7a1b355e49b342f28957f5e04ae95b18 P b0bb3b26f9e54bc8b16d05572a939d7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0bb3b26f9e54bc8b16d05572a939d7d, State: Initialized, Role: FOLLOWER
I20260619 08:58:35.418393 14626 consensus_queue.cc:260] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43053 } } peers { permanent_uuid: "1d9e834a6cb74090b30ff7e5b108e954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44613 } } peers { permanent_uuid: "ba1b76511e9a401fad69d6e7175118cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } }
I20260619 08:58:35.418516 14628 consensus_queue.cc:260] T 7a1b355e49b342f28957f5e04ae95b18 P b0bb3b26f9e54bc8b16d05572a939d7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43053 } } peers { permanent_uuid: "1d9e834a6cb74090b30ff7e5b108e954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44613 } } peers { permanent_uuid: "ba1b76511e9a401fad69d6e7175118cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } }
I20260619 08:58:35.418674 14626 ts_tablet_manager.cc:1428] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:58:35.418720 14344 heartbeater.cc:503] Master 127.0.98.62:46369 was elected leader, sending a full tablet report...
I20260619 08:58:35.418764 14628 ts_tablet_manager.cc:1428] T 7a1b355e49b342f28957f5e04ae95b18 P b0bb3b26f9e54bc8b16d05572a939d7d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:58:35.418859 14478 heartbeater.cc:503] Master 127.0.98.62:46369 was elected leader, sending a full tablet report...
I20260619 08:58:35.419306 14627 tablet_bootstrap.cc:492] T 7a1b355e49b342f28957f5e04ae95b18 P ba1b76511e9a401fad69d6e7175118cb: No bootstrap required, opened a new log
I20260619 08:58:35.419385 14627 ts_tablet_manager.cc:1397] T 7a1b355e49b342f28957f5e04ae95b18 P ba1b76511e9a401fad69d6e7175118cb: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 08:58:35.420984 14627 raft_consensus.cc:348] T 7a1b355e49b342f28957f5e04ae95b18 P ba1b76511e9a401fad69d6e7175118cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43053 } } peers { permanent_uuid: "1d9e834a6cb74090b30ff7e5b108e954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44613 } } peers { permanent_uuid: "ba1b76511e9a401fad69d6e7175118cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } }
I20260619 08:58:35.421125 14627 raft_consensus.cc:374] T 7a1b355e49b342f28957f5e04ae95b18 P ba1b76511e9a401fad69d6e7175118cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:58:35.421149 14627 raft_consensus.cc:729] T 7a1b355e49b342f28957f5e04ae95b18 P ba1b76511e9a401fad69d6e7175118cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba1b76511e9a401fad69d6e7175118cb, State: Initialized, Role: FOLLOWER
I20260619 08:58:35.421236 14627 consensus_queue.cc:260] T 7a1b355e49b342f28957f5e04ae95b18 P ba1b76511e9a401fad69d6e7175118cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43053 } } peers { permanent_uuid: "1d9e834a6cb74090b30ff7e5b108e954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44613 } } peers { permanent_uuid: "ba1b76511e9a401fad69d6e7175118cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } }
I20260619 08:58:35.421445 14627 ts_tablet_manager.cc:1428] T 7a1b355e49b342f28957f5e04ae95b18 P ba1b76511e9a401fad69d6e7175118cb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:58:35.421489 14612 heartbeater.cc:503] Master 127.0.98.62:46369 was elected leader, sending a full tablet report...
W20260619 08:58:35.473491 14479 tablet.cc:2369] T 7a1b355e49b342f28957f5e04ae95b18 P b0bb3b26f9e54bc8b16d05572a939d7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:58:35.508927 14632 raft_consensus.cc:482] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:58:35.509094 14632 raft_consensus.cc:504] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43053 } } peers { permanent_uuid: "1d9e834a6cb74090b30ff7e5b108e954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44613 } } peers { permanent_uuid: "ba1b76511e9a401fad69d6e7175118cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } }
I20260619 08:58:35.509466 14632 leader_election.cc:288] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0bb3b26f9e54bc8b16d05572a939d7d (127.0.98.2:43053), ba1b76511e9a401fad69d6e7175118cb (127.0.98.3:33255)
I20260619 08:58:35.512167 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:35.509786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44437 (local address 127.0.98.3:33255)
0619 08:58:35.509835 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:35.509837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:35.509844 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:58:35.509871 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:35.509872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:35.509881 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:35.509926 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:58:35.510079 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.510552 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.511462 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.511826 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.511886 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:35.511897 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:35.512044 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 08:58:35.512103 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 08:58:35.512167 14346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:35.509658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33255 (local address 127.0.98.1:44437)
0619 08:58:35.509718 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:35.509731 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:35.509795 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:35.509939 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:35.509940 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:35.509942 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:58:35.510018 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:35.510019 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.510634 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.510635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:35.511372 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:35.511374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.511898 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.511899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:35.511980 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:35.511983 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:58:35.512028 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 08:58:35.512068 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 08:58:35.512246 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:35.509764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55681 (local address 127.0.98.2:43053)
0619 08:58:35.509810 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:35.509812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:35.509921 (+ 109us) server_negotiation.cc:407] Connection header received
0619 08:58:35.509965 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:35.509966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:35.509979 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:35.510037 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:58:35.510173 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.510639 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.511538 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.511895 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.511952 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:35.511962 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:35.512140 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 08:58:35.512178 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 08:58:35.512272 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:35.509658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43053 (local address 127.0.98.1:55681)
0619 08:58:35.509848 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:35.509857 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:35.509916 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:35.510044 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:35.510045 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:35.510046 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:58:35.510114 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:35.510116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.510717 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.510718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:35.511452 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:35.511454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.511969 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.511970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:35.512047 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:35.512051 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:58:35.512131 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 08:58:35.512171 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":14,"thread_start_us":108,"threads_started":1}
I20260619 08:58:35.512599 14432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1b355e49b342f28957f5e04ae95b18" candidate_uuid: "1d9e834a6cb74090b30ff7e5b108e954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" is_pre_election: true
I20260619 08:58:35.512609 14566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1b355e49b342f28957f5e04ae95b18" candidate_uuid: "1d9e834a6cb74090b30ff7e5b108e954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba1b76511e9a401fad69d6e7175118cb" is_pre_election: true
I20260619 08:58:35.512728 14566 raft_consensus.cc:2454] T 7a1b355e49b342f28957f5e04ae95b18 P ba1b76511e9a401fad69d6e7175118cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d9e834a6cb74090b30ff7e5b108e954 in term 0.
I20260619 08:58:35.512734 14432 raft_consensus.cc:2454] T 7a1b355e49b342f28957f5e04ae95b18 P b0bb3b26f9e54bc8b16d05572a939d7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d9e834a6cb74090b30ff7e5b108e954 in term 0.
I20260619 08:58:35.512938 14232 leader_election.cc:302] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d9e834a6cb74090b30ff7e5b108e954, b0bb3b26f9e54bc8b16d05572a939d7d; no voters:
I20260619 08:58:35.513072 14632 raft_consensus.cc:2781] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:58:35.513130 14632 raft_consensus.cc:482] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:58:35.513147 14632 raft_consensus.cc:3037] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:58:35.513821 14632 raft_consensus.cc:504] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43053 } } peers { permanent_uuid: "1d9e834a6cb74090b30ff7e5b108e954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44613 } } peers { permanent_uuid: "ba1b76511e9a401fad69d6e7175118cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } }
I20260619 08:58:35.513944 14632 leader_election.cc:288] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954 [CANDIDATE]: Term 1 election: Requested vote from peers b0bb3b26f9e54bc8b16d05572a939d7d (127.0.98.2:43053), ba1b76511e9a401fad69d6e7175118cb (127.0.98.3:33255)
I20260619 08:58:35.514092 14566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1b355e49b342f28957f5e04ae95b18" candidate_uuid: "1d9e834a6cb74090b30ff7e5b108e954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba1b76511e9a401fad69d6e7175118cb"
I20260619 08:58:35.514110 14432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1b355e49b342f28957f5e04ae95b18" candidate_uuid: "1d9e834a6cb74090b30ff7e5b108e954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d"
I20260619 08:58:35.514171 14566 raft_consensus.cc:3037] T 7a1b355e49b342f28957f5e04ae95b18 P ba1b76511e9a401fad69d6e7175118cb [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:58:35.514174 14432 raft_consensus.cc:3037] T 7a1b355e49b342f28957f5e04ae95b18 P b0bb3b26f9e54bc8b16d05572a939d7d [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:58:35.515010 14566 raft_consensus.cc:2454] T 7a1b355e49b342f28957f5e04ae95b18 P ba1b76511e9a401fad69d6e7175118cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d9e834a6cb74090b30ff7e5b108e954 in term 1.
I20260619 08:58:35.515010 14432 raft_consensus.cc:2454] T 7a1b355e49b342f28957f5e04ae95b18 P b0bb3b26f9e54bc8b16d05572a939d7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d9e834a6cb74090b30ff7e5b108e954 in term 1.
I20260619 08:58:35.515168 14234 leader_election.cc:302] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d9e834a6cb74090b30ff7e5b108e954, ba1b76511e9a401fad69d6e7175118cb; no voters:
I20260619 08:58:35.515296 14632 raft_consensus.cc:2781] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:58:35.515457 14632 raft_consensus.cc:686] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954 [term 1 LEADER]: Becoming Leader. State: Replica: 1d9e834a6cb74090b30ff7e5b108e954, State: Running, Role: LEADER
I20260619 08:58:35.515543 14632 consensus_queue.cc:237] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43053 } } peers { permanent_uuid: "1d9e834a6cb74090b30ff7e5b108e954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44613 } } peers { permanent_uuid: "ba1b76511e9a401fad69d6e7175118cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } }
I20260619 08:58:35.516328 14153 catalog_manager.cc:5495] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d9e834a6cb74090b30ff7e5b108e954 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "1d9e834a6cb74090b30ff7e5b108e954" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d9e834a6cb74090b30ff7e5b108e954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba1b76511e9a401fad69d6e7175118cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } health_report { overall_health: UNKNOWN } } }
I20260619 08:58:35.523846 14624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:35.521831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52668 (local address 127.0.98.1:44613)
0619 08:58:35.521870 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:35.521872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:35.521880 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:58:35.521936 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:35.521937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:35.521953 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:35.522012 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:58:35.522107 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.522540 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.523221 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.523388 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.523430 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:35.523600 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:58:35.523691 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:58:35.523745 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:35.523781 (+ 36us) server_negotiation.cc:299] Negotiation successful
0619 08:58:35.523796 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 08:58:35.525441 14566 raft_consensus.cc:1261] T 7a1b355e49b342f28957f5e04ae95b18 P ba1b76511e9a401fad69d6e7175118cb [term 1 FOLLOWER]: Refusing update from remote peer 1d9e834a6cb74090b30ff7e5b108e954: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:58:35.525445 14432 raft_consensus.cc:1261] T 7a1b355e49b342f28957f5e04ae95b18 P b0bb3b26f9e54bc8b16d05572a939d7d [term 1 FOLLOWER]: Refusing update from remote peer 1d9e834a6cb74090b30ff7e5b108e954: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:58:35.525750 14632 consensus_queue.cc:1035] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba1b76511e9a401fad69d6e7175118cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:58:35.525895 14636 consensus_queue.cc:1035] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:58:35.528967 14640 mvcc.cc:205] Tried to move back new op lower bound from 7298496575589560320 to 7298496575551971328. Current Snapshot: MvccSnapshot[applied={T|T < 7298496575589560320 or (T in {7298496575589560320})}]
I20260619 08:58:35.530279 14152 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:58:35.530382 14152 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:58:35.531077 14152 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593: Sending DeleteTablet for 3 replicas of tablet 7a1b355e49b342f28957f5e04ae95b18
I20260619 08:58:35.531396 14412 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a1b355e49b342f28957f5e04ae95b18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:58:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40930
I20260619 08:58:35.531447 14278 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a1b355e49b342f28957f5e04ae95b18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:58:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52662
I20260619 08:58:35.531512 14546 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a1b355e49b342f28957f5e04ae95b18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:58:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33846
I20260619 08:58:35.531622 14648 tablet_replica.cc:330] T 7a1b355e49b342f28957f5e04ae95b18 P b0bb3b26f9e54bc8b16d05572a939d7d: stopping tablet replica
I20260619 08:58:35.531713 14648 raft_consensus.cc:2229] T 7a1b355e49b342f28957f5e04ae95b18 P b0bb3b26f9e54bc8b16d05572a939d7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:58:35.531778 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 14215
I20260619 08:58:35.531818 14648 raft_consensus.cc:2258] T 7a1b355e49b342f28957f5e04ae95b18 P b0bb3b26f9e54bc8b16d05572a939d7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:58:35.531785 14650 tablet_replica.cc:330] T 7a1b355e49b342f28957f5e04ae95b18 P ba1b76511e9a401fad69d6e7175118cb: stopping tablet replica
I20260619 08:58:35.531960 14650 raft_consensus.cc:2229] T 7a1b355e49b342f28957f5e04ae95b18 P ba1b76511e9a401fad69d6e7175118cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:58:35.532063 14650 raft_consensus.cc:2258] T 7a1b355e49b342f28957f5e04ae95b18 P ba1b76511e9a401fad69d6e7175118cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:58:35.532582 14650 ts_tablet_manager.cc:1905] T 7a1b355e49b342f28957f5e04ae95b18 P ba1b76511e9a401fad69d6e7175118cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:58:35.533005 14648 ts_tablet_manager.cc:1905] T 7a1b355e49b342f28957f5e04ae95b18 P b0bb3b26f9e54bc8b16d05572a939d7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:58:35.534322 14650 ts_tablet_manager.cc:1918] T 7a1b355e49b342f28957f5e04ae95b18 P ba1b76511e9a401fad69d6e7175118cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:58:35.534412 14650 log.cc:1196] T 7a1b355e49b342f28957f5e04ae95b18 P ba1b76511e9a401fad69d6e7175118cb: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/wal/wals/7a1b355e49b342f28957f5e04ae95b18
I20260619 08:58:35.534528 14648 ts_tablet_manager.cc:1918] T 7a1b355e49b342f28957f5e04ae95b18 P b0bb3b26f9e54bc8b16d05572a939d7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:58:35.534592 14648 log.cc:1196] T 7a1b355e49b342f28957f5e04ae95b18 P b0bb3b26f9e54bc8b16d05572a939d7d: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/wal/wals/7a1b355e49b342f28957f5e04ae95b18
I20260619 08:58:35.534677 14650 ts_tablet_manager.cc:1939] T 7a1b355e49b342f28957f5e04ae95b18 P ba1b76511e9a401fad69d6e7175118cb: Deleting consensus metadata
I20260619 08:58:35.534822 14648 ts_tablet_manager.cc:1939] T 7a1b355e49b342f28957f5e04ae95b18 P b0bb3b26f9e54bc8b16d05572a939d7d: Deleting consensus metadata
I20260619 08:58:35.535091 14140 catalog_manager.cc:4841] TS ba1b76511e9a401fad69d6e7175118cb (127.0.98.3:33255): tablet 7a1b355e49b342f28957f5e04ae95b18 (table test-table [id=635e7250c9c844bcb82097e30da5c2e7]) successfully deleted
I20260619 08:58:35.535135 14138 catalog_manager.cc:4841] TS b0bb3b26f9e54bc8b16d05572a939d7d (127.0.98.2:43053): tablet 7a1b355e49b342f28957f5e04ae95b18 (table test-table [id=635e7250c9c844bcb82097e30da5c2e7]) successfully deleted
W20260619 08:58:35.537335 14138 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44613 (error 108)
I20260619 08:58:35.537448 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 14349
I20260619 08:58:35.538189 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:35.538053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44613 (local address 127.0.0.1:52672)
0619 08:58:35.538110 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:35.538151 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 08:58:35.538323 14138 catalog_manager.cc:4584] TS 1d9e834a6cb74090b30ff7e5b108e954 (127.0.98.1:44613): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7a1b355e49b342f28957f5e04ae95b18: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44613: connect: Connection refused (error 111)
I20260619 08:58:35.541914 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 14483
I20260619 08:58:35.545895 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 14121
I20260619 08:58:35.550346 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46369
--webserver_interface=127.0.98.62
--webserver_port=34429
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:58:35.628507 14652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:58:35.628702 14652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:58:35.628722 14652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:58:35.629846 14652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:58:35.629889 14652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:58:35.629904 14652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:58:35.629917 14652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:58:35.629927 14652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:58:35.631372 14652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:58:35.631592 14652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:58:35.631804 14652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:58:35.639796 14658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:35.639816 14659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:35.639796 14661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:35.640108 14652 server_base.cc:1034] running on GCE node
I20260619 08:58:35.640388 14652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:58:35.640651 14652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:58:35.641819 14652 hybrid_clock.cc:648] HybridClock initialized: now 1781859515641792 us; error 35 us; skew 500 ppm
Jun 19 08:58:35 dist-test-slave-h4g5 krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:58:35.644367 14652 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 08:58:35.644802 14652 webserver.cc:460] Webserver started at http://127.0.98.62:34429/ using document root <none> and password file <none>
I20260619 08:58:35.645017 14652 fs_manager.cc:359] Metadata directory not provided
I20260619 08:58:35.645079 14652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:58:35.646399 14652 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:58:35.646958 14668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:35.647095 14652 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:35.647177 14652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "8991bbb9aea44da8a3e5b238d53a7593"
format_stamp: "Formatted at 2026-06-19 08:58:34 on dist-test-slave-h4g5"
server_key: "5a56c1513c720f332d576105d2e023d3"
server_key_iv: "3de4fa3ad67e8e0a96dab081147f7383"
server_key_version: "encryptionkey@0"
I20260619 08:58:35.647598 14652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:58:35.680379 14652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:58:35.682605 14652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:58:35.682747 14652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:58:35.686511 14652 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46369
I20260619 08:58:35.686651 14720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46369 every 8 connection(s)
I20260619 08:58:35.686910 14652 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:58:35.688597 14721 sys_catalog.cc:263] Verifying existing consensus state
I20260619 08:58:35.689193 14721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593: Bootstrap starting.
I20260619 08:58:35.695041 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 14652
I20260619 08:58:35.695328 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44613
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46369
--webserver_port=45279
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 08:58:35.697894 14721 log.cc:824] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593: Log is configured to *not* fsync() on all Append() calls
I20260619 08:58:35.699995 14721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 08:58:35.700290 14721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593: Bootstrap complete.
I20260619 08:58:35.702438 14721 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8991bbb9aea44da8a3e5b238d53a7593" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46369 } }
I20260619 08:58:35.702679 14721 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8991bbb9aea44da8a3e5b238d53a7593, State: Initialized, Role: FOLLOWER
I20260619 08:58:35.702828 14721 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8991bbb9aea44da8a3e5b238d53a7593" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46369 } }
I20260619 08:58:35.702900 14721 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:58:35.702953 14721 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:58:35.703001 14721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [term 1 FOLLOWER]: Advancing to term 2
I20260619 08:58:35.704042 14721 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8991bbb9aea44da8a3e5b238d53a7593" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46369 } }
I20260619 08:58:35.704175 14721 leader_election.cc:302] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8991bbb9aea44da8a3e5b238d53a7593; no voters:
I20260619 08:58:35.704344 14721 leader_election.cc:288] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 08:58:35.704391 14725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 08:58:35.704537 14725 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [term 2 LEADER]: Becoming Leader. State: Replica: 8991bbb9aea44da8a3e5b238d53a7593, State: Running, Role: LEADER
I20260619 08:58:35.704625 14721 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:58:35.704648 14725 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8991bbb9aea44da8a3e5b238d53a7593" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46369 } }
I20260619 08:58:35.704907 14725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8991bbb9aea44da8a3e5b238d53a7593. Latest consensus state: current_term: 2 leader_uuid: "8991bbb9aea44da8a3e5b238d53a7593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8991bbb9aea44da8a3e5b238d53a7593" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46369 } } }
I20260619 08:58:35.704973 14725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [sys.catalog]: This master's current role is: LEADER
I20260619 08:58:35.705112 14726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8991bbb9aea44da8a3e5b238d53a7593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8991bbb9aea44da8a3e5b238d53a7593" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46369 } } }
I20260619 08:58:35.705178 14726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [sys.catalog]: This master's current role is: LEADER
I20260619 08:58:35.705435 14730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:58:35.706228 14730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:58:35.706331 14730 catalog_manager.cc:1259] Loaded cluster ID: e8c4656838104c6894f1c2d9db3427b1
I20260619 08:58:35.706365 14730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:58:35.706851 14730 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 08:58:35.711449 14730 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 08:58:35.713907 14730 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:58:35.714110 14730 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593: Loaded TSK: 0
I20260619 08:58:35.714320 14730 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 08:58:35.776465 14723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:58:35.776672 14723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:58:35.776695 14723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:58:35.777835 14723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:58:35.777878 14723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:58:35.777904 14723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 08:58:35.779331 14723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44613
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45279
--tserver_master_addrs=127.0.98.62:46369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:58:35.779561 14723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:58:35.779774 14723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:58:35.784658 14748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:35.784706 14749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:35.784837 14723 server_base.cc:1034] running on GCE node
W20260619 08:58:35.784992 14751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:35.785192 14723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:58:35.785413 14723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:58:35.786561 14723 hybrid_clock.cc:648] HybridClock initialized: now 1781859515786550 us; error 35 us; skew 500 ppm
Jun 19 08:58:35 dist-test-slave-h4g5 krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:58:35.789013 14723 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 08:58:35.789454 14723 webserver.cc:460] Webserver started at http://127.0.98.1:45279/ using document root <none> and password file <none>
I20260619 08:58:35.789676 14723 fs_manager.cc:359] Metadata directory not provided
I20260619 08:58:35.789738 14723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:58:35.791136 14723 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:58:35.791766 14758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:35.791929 14723 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:58:35.792027 14723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/wal
uuid: "1d9e834a6cb74090b30ff7e5b108e954"
format_stamp: "Formatted at 2026-06-19 08:58:35 on dist-test-slave-h4g5"
server_key: "a2290833a50be30419b4b21f5df4a2f9"
server_key_iv: "5c02e885c7370220c31eb5030831c96d"
server_key_version: "encryptionkey@0"
I20260619 08:58:35.792477 14723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:58:35.818849 14723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:58:35.821113 14723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:58:35.821255 14723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:58:35.821499 14723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:58:35.821992 14765 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 08:58:35.823446 14723 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 08:58:35.823503 14723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 08:58:35.823542 14723 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 08:58:35.824411 14723 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 08:58:35.824455 14723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260619 08:58:35.824565 14765 tablet_bootstrap.cc:492] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954: Bootstrap starting.
I20260619 08:58:35.831568 14723 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44613
I20260619 08:58:35.831826 14872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44613 every 8 connection(s)
I20260619 08:58:35.831967 14723 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
I20260619 08:58:35.834610 14765 log.cc:824] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954: Log is configured to *not* fsync() on all Append() calls
I20260619 08:58:35.837091 14765 tablet_bootstrap.cc:492] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 08:58:35.837343 14765 tablet_bootstrap.cc:492] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954: Bootstrap complete.
I20260619 08:58:35.838192 14765 ts_tablet_manager.cc:1397] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.010s
I20260619 08:58:35.839397 14765 raft_consensus.cc:348] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43053 } } peers { permanent_uuid: "1d9e834a6cb74090b30ff7e5b108e954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44613 } } peers { permanent_uuid: "ba1b76511e9a401fad69d6e7175118cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } }
I20260619 08:58:35.839627 14765 raft_consensus.cc:729] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d9e834a6cb74090b30ff7e5b108e954, State: Initialized, Role: FOLLOWER
I20260619 08:58:35.839754 14765 consensus_queue.cc:260] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43053 } } peers { permanent_uuid: "1d9e834a6cb74090b30ff7e5b108e954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44613 } } peers { permanent_uuid: "ba1b76511e9a401fad69d6e7175118cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } }
I20260619 08:58:35.840006 14765 ts_tablet_manager.cc:1428] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
Jun 19 08:58:35 dist-test-slave-h4g5 krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:58:35.840777 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 14723
I20260619 08:58:35.841048 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43053
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46369
--webserver_port=34677
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 08:58:35.844286 14876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:35.832828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33359 (local address 127.0.98.62:46369)
0619 08:58:35.833019 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:35.833029 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:35.833254 (+ 225us) server_negotiation.cc:407] Connection header received
0619 08:58:35.834082 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:35.834089 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:35.834144 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:35.834297 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:35.835068 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.836472 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.837883 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.838475 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.838594 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:35.840980 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:35.840983 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:35.840983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:35.840993 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:35.842490 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:35.842986 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:35.842989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:35.842989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:35.843096 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:35.843356 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:35.843357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:35.843358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:35.843582 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:35.843653 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:35.843928 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 08:58:35.844059 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":92,"threads_started":1}
I20260619 08:58:35.844286 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:35.832708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.1:33359)
0619 08:58:35.833068 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:35.833132 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:35.833832 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:35.834341 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:35.834346 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:35.834701 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:58:35.834946 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:35.834952 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.836609 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.836610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:35.837807 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:35.837810 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.838602 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.838603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:35.838719 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:35.839255 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:58:35.839263 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:58:35.840839 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:58:35.842568 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:35.842574 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:35.842576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:35.842895 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:35.843181 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:35.843182 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:35.843183 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:35.843298 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:35.843647 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:58:35.843653 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:58:35.843781 (+ 128us) client_negotiation.cc:769] Sending connection context
0619 08:58:35.843867 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 08:58:35.844061 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":88,"threads_started":1}
I20260619 08:58:35.844935 14873 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46369
I20260619 08:58:35.845041 14873 heartbeater.cc:464] Registering TS with master...
I20260619 08:58:35.845230 14873 heartbeater.cc:511] Master 127.0.98.62:46369 requested a full tablet report, sending...
I20260619 08:58:35.845924 14685 ts_manager.cc:194] Registered new tserver with Master: 1d9e834a6cb74090b30ff7e5b108e954 (127.0.98.1:44613)
I20260619 08:58:35.846948 14685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33359
I20260619 08:58:35.847865 14873 heartbeater.cc:503] Master 127.0.98.62:46369 was elected leader, sending a full tablet report...
I20260619 08:58:35.854683 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:35.846469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44613 (local address 127.0.0.1:52684)
0619 08:58:35.846680 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:35.846710 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:35.846764 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:35.851405 (+ 4641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:35.851407 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:35.851410 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:58:35.851467 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:35.851469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.852718 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.852719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:35.853505 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:35.853507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.854425 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.854426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:35.854495 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:35.854507 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 08:58:35.854578 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 08:58:35.854623 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":95,"threads_started":1}
I20260619 08:58:35.854712 14881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:35.846542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52684 (local address 127.0.98.1:44613)
0619 08:58:35.851241 (+ 4699us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:35.851246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:35.851257 (+ 11us) server_negotiation.cc:407] Connection header received
0619 08:58:35.851291 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:35.851294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:35.851308 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:35.851398 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:58:35.851529 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.852647 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.853584 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.854349 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.854417 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:35.854472 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:35.854601 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 08:58:35.854638 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4665,"thread_start_us":86,"threads_started":1}
I20260619 08:58:35.855075 14807 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a1b355e49b342f28957f5e04ae95b18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:58:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52684
I20260619 08:58:35.855563 14882 tablet_replica.cc:330] stopping tablet replica
I20260619 08:58:35.855654 14882 raft_consensus.cc:2229] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:58:35.855721 14882 raft_consensus.cc:2258] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:58:35.856007 14882 ts_tablet_manager.cc:1905] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:58:35.858084 14882 ts_tablet_manager.cc:1918] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:58:35.858144 14882 log.cc:1196] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/wal/wals/7a1b355e49b342f28957f5e04ae95b18
I20260619 08:58:35.858325 14882 ts_tablet_manager.cc:1939] T 7a1b355e49b342f28957f5e04ae95b18 P 1d9e834a6cb74090b30ff7e5b108e954: Deleting consensus metadata
I20260619 08:58:35.858799 14670 catalog_manager.cc:4841] TS 1d9e834a6cb74090b30ff7e5b108e954 (127.0.98.1:44613): tablet 7a1b355e49b342f28957f5e04ae95b18 (table test-table [id=635e7250c9c844bcb82097e30da5c2e7]) successfully deleted
W20260619 08:58:35.921362 14879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:58:35.921563 14879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:58:35.921594 14879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:58:35.922719 14879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:58:35.922757 14879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:58:35.922791 14879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 08:58:35.924248 14879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34677
--tserver_master_addrs=127.0.98.62:46369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:58:35.924474 14879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:58:35.924675 14879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:58:35.929359 14887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:35.929301 14890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:35.929345 14879 server_base.cc:1034] running on GCE node
W20260619 08:58:35.929311 14888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:35.929747 14879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:58:35.929976 14879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:58:35.931115 14879 hybrid_clock.cc:648] HybridClock initialized: now 1781859515931110 us; error 33 us; skew 500 ppm
Jun 19 08:58:35 dist-test-slave-h4g5 krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:58:35.933380 14879 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 08:58:35.933789 14879 webserver.cc:460] Webserver started at http://127.0.98.2:34677/ using document root <none> and password file <none>
I20260619 08:58:35.933990 14879 fs_manager.cc:359] Metadata directory not provided
I20260619 08:58:35.934049 14879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:58:35.935286 14879 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:58:35.935786 14897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:35.935926 14879 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:35.936017 14879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/wal
uuid: "b0bb3b26f9e54bc8b16d05572a939d7d"
format_stamp: "Formatted at 2026-06-19 08:58:35 on dist-test-slave-h4g5"
server_key: "caf54e9d0162b428c8b0e0f052afde4d"
server_key_iv: "e2af6d8a5ebb5c02ff8bc957fce5b335"
server_key_version: "encryptionkey@0"
I20260619 08:58:35.936355 14879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:58:35.959874 14879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:58:35.961622 14879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:58:35.961756 14879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:58:35.961978 14879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:58:35.962316 14879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:58:35.962357 14879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:35.962390 14879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:58:35.962406 14879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:35.967905 14879 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43053
I20260619 08:58:35.967963 15010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43053 every 8 connection(s)
I20260619 08:58:35.968300 14879 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 08:58:35 dist-test-slave-h4g5 krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:58:35.975778 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 14879
I20260619 08:58:35.976023 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33255
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46369
--webserver_port=32985
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 08:58:35.976888 14876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:35.968959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41217 (local address 127.0.98.62:46369)
0619 08:58:35.969021 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:35.969022 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:35.969216 (+ 194us) server_negotiation.cc:407] Connection header received
0619 08:58:35.969694 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:35.969695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:35.969707 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:35.969742 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:35.970656 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.971140 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.972133 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.972346 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.972398 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:35.974290 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:35.974291 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:35.974292 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:35.974299 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:35.975501 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:35.975903 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:35.975905 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:35.975905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:35.975965 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:35.976272 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:35.976273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:35.976273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:35.976386 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:35.976454 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:35.976696 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 08:58:35.976800 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 08:58:35.977051 15013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:35.968943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.2:41217)
0619 08:58:35.969101 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:35.969136 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:35.969602 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:35.969804 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:35.969807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:35.970389 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:58:35.970579 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:35.970584 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.971210 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.971211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:35.972025 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:35.972027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:35.972409 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:35.972410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:35.972505 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:35.972926 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:58:35.972934 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:58:35.974211 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:58:35.975585 (+ 1374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:35.975591 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:35.975593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:35.975804 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:35.976025 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:35.976026 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:35.976026 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:35.976172 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:35.976446 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:58:35.976452 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:58:35.976582 (+ 130us) client_negotiation.cc:769] Sending connection context
0619 08:58:35.976669 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 08:58:35.976819 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":95,"threads_started":1}
I20260619 08:58:35.977522 15011 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46369
I20260619 08:58:35.977630 15011 heartbeater.cc:464] Registering TS with master...
I20260619 08:58:35.977813 15011 heartbeater.cc:511] Master 127.0.98.62:46369 requested a full tablet report, sending...
I20260619 08:58:35.978189 14685 ts_manager.cc:194] Registered new tserver with Master: b0bb3b26f9e54bc8b16d05572a939d7d (127.0.98.2:43053)
I20260619 08:58:35.978549 14685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41217
W20260619 08:58:36.056283 15014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:58:36.056511 15014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:58:36.056535 15014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:58:36.057699 15014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:58:36.057744 15014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:58:36.057770 15014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 08:58:36.059206 15014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=32985
--tserver_master_addrs=127.0.98.62:46369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:58:36.059450 15014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:58:36.059665 15014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:58:36.064716 15020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:36.064709 15022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:36.064709 15019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:36.064942 15014 server_base.cc:1034] running on GCE node
I20260619 08:58:36.065205 15014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:58:36.065455 15014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:58:36.066599 15014 hybrid_clock.cc:648] HybridClock initialized: now 1781859516066585 us; error 28 us; skew 500 ppm
Jun 19 08:58:36 dist-test-slave-h4g5 krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:58:36.069006 15014 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 08:58:36.069463 15014 webserver.cc:460] Webserver started at http://127.0.98.3:32985/ using document root <none> and password file <none>
I20260619 08:58:36.069665 15014 fs_manager.cc:359] Metadata directory not provided
I20260619 08:58:36.069729 15014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:58:36.071061 15014 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:58:36.071593 15029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:36.071743 15014 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:58:36.071823 15014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/wal
uuid: "ba1b76511e9a401fad69d6e7175118cb"
format_stamp: "Formatted at 2026-06-19 08:58:35 on dist-test-slave-h4g5"
server_key: "24e860de45ea5d1e84425175fae14ea6"
server_key_iv: "03c19c7894473b74e1e493c956ba66cb"
server_key_version: "encryptionkey@0"
I20260619 08:58:36.072182 15014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:58:36.097481 15014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:58:36.099190 15014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:58:36.099323 15014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:58:36.099543 15014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:58:36.099822 15014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:58:36.099854 15014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:36.099884 15014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:58:36.099906 15014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:36.105436 15014 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33255
I20260619 08:58:36.105471 15142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33255 every 8 connection(s)
I20260619 08:58:36.105931 15014 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
I20260619 08:58:36.111155 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 15014
Jun 19 08:58:36 dist-test-slave-h4g5 krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:58:36.116868 14876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:36.106741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54179 (local address 127.0.98.62:46369)
0619 08:58:36.106863 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:36.106865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:36.107568 (+ 703us) server_negotiation.cc:407] Connection header received
0619 08:58:36.108186 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:36.108188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:36.108202 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:36.108247 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:36.109188 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:36.109685 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:36.111079 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:36.111313 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:36.111423 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:36.114047 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:36.114049 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:36.114049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:36.114057 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:36.115439 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:36.115860 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:36.115860 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:36.115861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:36.115932 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:36.116180 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:36.116181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:36.116182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:36.116293 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:36.116352 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:36.116653 (+ 301us) server_negotiation.cc:299] Negotiation successful
0619 08:58:36.116776 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 08:58:36.116982 15145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:36.106694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.3:54179)
0619 08:58:36.107446 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:36.107487 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:36.108070 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:36.108319 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:36.108323 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:36.108886 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:58:36.109092 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:36.109096 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:36.109767 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:36.109768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:36.110976 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:36.110979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:36.111427 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:36.111428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:36.111572 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:36.112197 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:58:36.112208 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:58:36.113960 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:58:36.115518 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:36.115524 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:36.115527 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:36.115801 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:36.115990 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:36.115990 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:36.115990 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:36.116115 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:36.116350 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:58:36.116356 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:58:36.116522 (+ 166us) client_negotiation.cc:769] Sending connection context
0619 08:58:36.116604 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 08:58:36.116762 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":80,"threads_started":1}
I20260619 08:58:36.117424 15143 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46369
I20260619 08:58:36.117529 15143 heartbeater.cc:464] Registering TS with master...
I20260619 08:58:36.117722 15143 heartbeater.cc:511] Master 127.0.98.62:46369 requested a full tablet report, sending...
I20260619 08:58:36.118144 14685 ts_manager.cc:194] Registered new tserver with Master: ba1b76511e9a401fad69d6e7175118cb (127.0.98.3:33255)
I20260619 08:58:36.118351 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:36.111507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47886 (local address 127.0.98.62:46369)
0619 08:58:36.111878 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:36.111880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:36.111887 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:58:36.111908 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:36.111908 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:36.111920 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:36.111980 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:36.112712 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:36.113367 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:36.114272 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:36.114531 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:36.114595 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:36.115831 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:36.115832 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:36.115832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:36.115839 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:36.117241 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:36.117647 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:36.117647 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:36.117648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:36.117699 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:36.117876 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:36.117876 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:36.117877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:36.117997 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:36.118148 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:36.118185 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 08:58:36.118260 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":72,"threads_started":1}
I20260619 08:58:36.118557 14685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54179
I20260619 08:58:36.119025 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:58:36.121593 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:36.119325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47900 (local address 127.0.98.62:46369)
0619 08:58:36.119359 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:36.119361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:36.119461 (+ 100us) server_negotiation.cc:407] Connection header received
0619 08:58:36.119522 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:36.119522 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:36.119566 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:36.119623 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:58:36.119720 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:36.120215 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:36.120920 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:36.121121 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:36.121171 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:36.121312 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:58:36.121405 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:58:36.121469 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:36.121504 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 08:58:36.121525 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 08:58:36.121824 14685 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:47900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:58:36.123296 14685 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:58:36.125231 14807 tablet_service.cc:1467] Processing CreateTablet for tablet 7349478f688042e3b70a0a23aaf3e8ce (DEFAULT_TABLE table=test-table [id=0aa7a79a390f4b0aa7a0645f62a294e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:58:36.125424 14807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7349478f688042e3b70a0a23aaf3e8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:58:36.126869 14765 tablet_bootstrap.cc:492] T 7349478f688042e3b70a0a23aaf3e8ce P 1d9e834a6cb74090b30ff7e5b108e954: Bootstrap starting.
I20260619 08:58:36.127187 14765 tablet_bootstrap.cc:654] T 7349478f688042e3b70a0a23aaf3e8ce P 1d9e834a6cb74090b30ff7e5b108e954: Neither blocks nor log segments found. Creating new log.
I20260619 08:58:36.127622 14765 tablet_bootstrap.cc:492] T 7349478f688042e3b70a0a23aaf3e8ce P 1d9e834a6cb74090b30ff7e5b108e954: No bootstrap required, opened a new log
I20260619 08:58:36.127661 14765 ts_tablet_manager.cc:1397] T 7349478f688042e3b70a0a23aaf3e8ce P 1d9e834a6cb74090b30ff7e5b108e954: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260619 08:58:36.127758 14765 raft_consensus.cc:348] T 7349478f688042e3b70a0a23aaf3e8ce P 1d9e834a6cb74090b30ff7e5b108e954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1b76511e9a401fad69d6e7175118cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } } peers { permanent_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43053 } } peers { permanent_uuid: "1d9e834a6cb74090b30ff7e5b108e954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44613 } }
I20260619 08:58:36.127817 14765 raft_consensus.cc:374] T 7349478f688042e3b70a0a23aaf3e8ce P 1d9e834a6cb74090b30ff7e5b108e954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:58:36.127867 14765 raft_consensus.cc:729] T 7349478f688042e3b70a0a23aaf3e8ce P 1d9e834a6cb74090b30ff7e5b108e954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d9e834a6cb74090b30ff7e5b108e954, State: Initialized, Role: FOLLOWER
I20260619 08:58:36.127910 14765 consensus_queue.cc:260] T 7349478f688042e3b70a0a23aaf3e8ce P 1d9e834a6cb74090b30ff7e5b108e954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1b76511e9a401fad69d6e7175118cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } } peers { permanent_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43053 } } peers { permanent_uuid: "1d9e834a6cb74090b30ff7e5b108e954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44613 } }
I20260619 08:58:36.127991 14765 ts_tablet_manager.cc:1428] T 7349478f688042e3b70a0a23aaf3e8ce P 1d9e834a6cb74090b30ff7e5b108e954: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:36.128755 15149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:36.124818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43053 (local address 127.0.0.1:40944)
0619 08:58:36.125151 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:36.125160 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:36.125223 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:36.125451 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:36.125452 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:36.125454 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:58:36.125496 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:36.125497 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:36.126493 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:36.126493 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:36.127590 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:36.127592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:36.128480 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:36.128480 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:36.128571 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:36.128575 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:58:36.128631 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 08:58:36.128673 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":18,"thread_start_us":103,"threads_started":1}
I20260619 08:58:36.128834 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:36.124817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33255 (local address 127.0.0.1:33848)
0619 08:58:36.124867 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:36.124876 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:36.124911 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:36.125486 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:36.125487 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:36.125488 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:58:36.125513 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:36.125514 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:36.126639 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:36.126639 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:36.127676 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:36.127678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:36.128546 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:36.128551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:36.128617 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:36.128621 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:58:36.128647 (+ 26us) client_negotiation.cc:240] Negotiation successful
0619 08:58:36.128695 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 08:58:36.129016 15150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:36.124905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33848 (local address 127.0.98.3:33255)
0619 08:58:36.125320 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:36.125324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:36.125333 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:58:36.125366 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:36.125367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:36.125382 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:36.125458 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:58:36.125569 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:36.126527 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:36.127730 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:36.128453 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:36.128795 (+ 342us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:36.128865 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:36.128893 (+ 28us) server_negotiation.cc:299] Negotiation successful
0619 08:58:36.128944 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":82,"threads_started":1}
I20260619 08:58:36.129043 15151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:36.124923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40944 (local address 127.0.98.2:43053)
0619 08:58:36.125094 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:36.125099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:36.125231 (+ 132us) server_negotiation.cc:407] Connection header received
0619 08:58:36.125275 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:36.125277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:36.125291 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:36.125362 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:58:36.125581 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:36.126405 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:36.127674 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:36.128411 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:36.128475 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:36.128532 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:36.128682 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 08:58:36.128726 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":71,"threads_started":1}
I20260619 08:58:36.129667 15077 tablet_service.cc:1467] Processing CreateTablet for tablet 7349478f688042e3b70a0a23aaf3e8ce (DEFAULT_TABLE table=test-table [id=0aa7a79a390f4b0aa7a0645f62a294e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:58:36.129670 14945 tablet_service.cc:1467] Processing CreateTablet for tablet 7349478f688042e3b70a0a23aaf3e8ce (DEFAULT_TABLE table=test-table [id=0aa7a79a390f4b0aa7a0645f62a294e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:58:36.129956 14945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7349478f688042e3b70a0a23aaf3e8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:58:36.129956 15077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7349478f688042e3b70a0a23aaf3e8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:58:36.132649 15154 tablet_bootstrap.cc:492] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d: Bootstrap starting.
I20260619 08:58:36.133267 15154 tablet_bootstrap.cc:654] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d: Neither blocks nor log segments found. Creating new log.
I20260619 08:58:36.133708 15154 log.cc:824] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d: Log is configured to *not* fsync() on all Append() calls
I20260619 08:58:36.134002 15153 tablet_bootstrap.cc:492] T 7349478f688042e3b70a0a23aaf3e8ce P ba1b76511e9a401fad69d6e7175118cb: Bootstrap starting.
I20260619 08:58:36.134330 15154 tablet_bootstrap.cc:492] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d: No bootstrap required, opened a new log
I20260619 08:58:36.134388 15154 ts_tablet_manager.cc:1397] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:58:36.134538 15153 tablet_bootstrap.cc:654] T 7349478f688042e3b70a0a23aaf3e8ce P ba1b76511e9a401fad69d6e7175118cb: Neither blocks nor log segments found. Creating new log.
I20260619 08:58:36.134931 15153 log.cc:824] T 7349478f688042e3b70a0a23aaf3e8ce P ba1b76511e9a401fad69d6e7175118cb: Log is configured to *not* fsync() on all Append() calls
I20260619 08:58:36.135510 15153 tablet_bootstrap.cc:492] T 7349478f688042e3b70a0a23aaf3e8ce P ba1b76511e9a401fad69d6e7175118cb: No bootstrap required, opened a new log
I20260619 08:58:36.135569 15153 ts_tablet_manager.cc:1397] T 7349478f688042e3b70a0a23aaf3e8ce P ba1b76511e9a401fad69d6e7175118cb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:58:36.135804 15154 raft_consensus.cc:348] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1b76511e9a401fad69d6e7175118cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } } peers { permanent_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43053 } } peers { permanent_uuid: "1d9e834a6cb74090b30ff7e5b108e954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44613 } }
I20260619 08:58:36.135944 15154 raft_consensus.cc:374] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:58:36.135975 15154 raft_consensus.cc:729] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0bb3b26f9e54bc8b16d05572a939d7d, State: Initialized, Role: FOLLOWER
I20260619 08:58:36.136111 15154 consensus_queue.cc:260] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1b76511e9a401fad69d6e7175118cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } } peers { permanent_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43053 } } peers { permanent_uuid: "1d9e834a6cb74090b30ff7e5b108e954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44613 } }
I20260619 08:58:36.136348 15154 ts_tablet_manager.cc:1428] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:58:36.136397 15011 heartbeater.cc:503] Master 127.0.98.62:46369 was elected leader, sending a full tablet report...
I20260619 08:58:36.137225 15153 raft_consensus.cc:348] T 7349478f688042e3b70a0a23aaf3e8ce P ba1b76511e9a401fad69d6e7175118cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1b76511e9a401fad69d6e7175118cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } } peers { permanent_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43053 } } peers { permanent_uuid: "1d9e834a6cb74090b30ff7e5b108e954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44613 } }
I20260619 08:58:36.137380 15153 raft_consensus.cc:374] T 7349478f688042e3b70a0a23aaf3e8ce P ba1b76511e9a401fad69d6e7175118cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:58:36.137416 15153 raft_consensus.cc:729] T 7349478f688042e3b70a0a23aaf3e8ce P ba1b76511e9a401fad69d6e7175118cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba1b76511e9a401fad69d6e7175118cb, State: Initialized, Role: FOLLOWER
I20260619 08:58:36.137521 15153 consensus_queue.cc:260] T 7349478f688042e3b70a0a23aaf3e8ce P ba1b76511e9a401fad69d6e7175118cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1b76511e9a401fad69d6e7175118cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } } peers { permanent_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43053 } } peers { permanent_uuid: "1d9e834a6cb74090b30ff7e5b108e954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44613 } }
I20260619 08:58:36.137754 15153 ts_tablet_manager.cc:1428] T 7349478f688042e3b70a0a23aaf3e8ce P ba1b76511e9a401fad69d6e7175118cb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:58:36.137773 15143 heartbeater.cc:503] Master 127.0.98.62:46369 was elected leader, sending a full tablet report...
W20260619 08:58:36.218820 15012 tablet.cc:2369] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:58:36.254868 15157 raft_consensus.cc:482] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:58:36.255002 15157 raft_consensus.cc:504] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1b76511e9a401fad69d6e7175118cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } } peers { permanent_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43053 } } peers { permanent_uuid: "1d9e834a6cb74090b30ff7e5b108e954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44613 } }
I20260619 08:58:36.255383 15157 leader_election.cc:288] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba1b76511e9a401fad69d6e7175118cb (127.0.98.3:33255), 1d9e834a6cb74090b30ff7e5b108e954 (127.0.98.1:44613)
I20260619 08:58:36.258072 15150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:36.255654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48433 (local address 127.0.98.3:33255)
0619 08:58:36.255697 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:36.255699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:36.255815 (+ 116us) server_negotiation.cc:407] Connection header received
0619 08:58:36.255872 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:36.255873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:36.255886 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:36.255933 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:58:36.256065 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:36.256518 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:36.257404 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:36.257760 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:36.257816 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:36.257826 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:36.257970 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 08:58:36.258006 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 08:58:36.258087 15013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:36.255569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44613 (local address 127.0.98.2:59113)
0619 08:58:36.255611 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:36.255624 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:36.255671 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:36.255840 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:36.255841 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:36.255843 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:58:36.255915 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:36.255916 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:36.256573 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:36.256574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:36.257306 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:36.257307 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:36.257831 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:36.257832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:36.257912 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:36.257915 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:58:36.257963 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 08:58:36.258001 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":1}
I20260619 08:58:36.258080 14881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:36.255667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59113 (local address 127.0.98.1:44613)
0619 08:58:36.255715 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:36.255717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:36.255728 (+ 11us) server_negotiation.cc:407] Connection header received
0619 08:58:36.255758 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:36.255759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:36.255771 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:36.255826 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:58:36.255982 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:36.256497 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:36.257394 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:36.257757 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:36.257816 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:36.257827 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:36.257976 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 08:58:36.258015 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 08:58:36.258108 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:36.255569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33255 (local address 127.0.98.2:48433)
0619 08:58:36.255745 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:36.255753 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:36.255815 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:36.255939 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:36.255940 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:36.255941 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:58:36.256008 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:36.256009 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:36.256598 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:36.256598 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:36.257318 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:36.257320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:36.257837 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:36.257838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:36.257901 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:36.257904 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:58:36.257946 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 08:58:36.257985 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":14,"thread_start_us":90,"threads_started":1}
I20260619 08:58:36.258384 15097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7349478f688042e3b70a0a23aaf3e8ce" candidate_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba1b76511e9a401fad69d6e7175118cb" is_pre_election: true
I20260619 08:58:36.258544 15097 raft_consensus.cc:2454] T 7349478f688042e3b70a0a23aaf3e8ce P ba1b76511e9a401fad69d6e7175118cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0bb3b26f9e54bc8b16d05572a939d7d in term 0.
I20260619 08:58:36.258580 14827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7349478f688042e3b70a0a23aaf3e8ce" candidate_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d9e834a6cb74090b30ff7e5b108e954" is_pre_election: true
I20260619 08:58:36.258726 14827 raft_consensus.cc:2454] T 7349478f688042e3b70a0a23aaf3e8ce P 1d9e834a6cb74090b30ff7e5b108e954 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0bb3b26f9e54bc8b16d05572a939d7d in term 0.
I20260619 08:58:36.258760 14901 leader_election.cc:302] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0bb3b26f9e54bc8b16d05572a939d7d, ba1b76511e9a401fad69d6e7175118cb; no voters:
I20260619 08:58:36.258879 15157 raft_consensus.cc:2781] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:58:36.258931 15157 raft_consensus.cc:482] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:58:36.258950 15157 raft_consensus.cc:3037] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:58:36.259611 15157 raft_consensus.cc:504] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1b76511e9a401fad69d6e7175118cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } } peers { permanent_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43053 } } peers { permanent_uuid: "1d9e834a6cb74090b30ff7e5b108e954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44613 } }
I20260619 08:58:36.259727 15157 leader_election.cc:288] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [CANDIDATE]: Term 1 election: Requested vote from peers ba1b76511e9a401fad69d6e7175118cb (127.0.98.3:33255), 1d9e834a6cb74090b30ff7e5b108e954 (127.0.98.1:44613)
I20260619 08:58:36.259976 15097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7349478f688042e3b70a0a23aaf3e8ce" candidate_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba1b76511e9a401fad69d6e7175118cb"
I20260619 08:58:36.259976 14827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7349478f688042e3b70a0a23aaf3e8ce" candidate_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d9e834a6cb74090b30ff7e5b108e954"
I20260619 08:58:36.260061 14827 raft_consensus.cc:3037] T 7349478f688042e3b70a0a23aaf3e8ce P 1d9e834a6cb74090b30ff7e5b108e954 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:58:36.260061 15097 raft_consensus.cc:3037] T 7349478f688042e3b70a0a23aaf3e8ce P ba1b76511e9a401fad69d6e7175118cb [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:58:36.260964 14827 raft_consensus.cc:2454] T 7349478f688042e3b70a0a23aaf3e8ce P 1d9e834a6cb74090b30ff7e5b108e954 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0bb3b26f9e54bc8b16d05572a939d7d in term 1.
I20260619 08:58:36.260964 15097 raft_consensus.cc:2454] T 7349478f688042e3b70a0a23aaf3e8ce P ba1b76511e9a401fad69d6e7175118cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0bb3b26f9e54bc8b16d05572a939d7d in term 1.
I20260619 08:58:36.261140 14899 leader_election.cc:302] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d9e834a6cb74090b30ff7e5b108e954, b0bb3b26f9e54bc8b16d05572a939d7d; no voters:
I20260619 08:58:36.261231 15157 raft_consensus.cc:2781] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:58:36.261379 15157 raft_consensus.cc:686] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [term 1 LEADER]: Becoming Leader. State: Replica: b0bb3b26f9e54bc8b16d05572a939d7d, State: Running, Role: LEADER
I20260619 08:58:36.261502 15157 consensus_queue.cc:237] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1b76511e9a401fad69d6e7175118cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } } peers { permanent_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43053 } } peers { permanent_uuid: "1d9e834a6cb74090b30ff7e5b108e954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44613 } }
I20260619 08:58:36.262346 14685 catalog_manager.cc:5495] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d reported cstate change: term changed from 0 to 1, leader changed from <none> to b0bb3b26f9e54bc8b16d05572a939d7d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1b76511e9a401fad69d6e7175118cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d9e834a6cb74090b30ff7e5b108e954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44613 } health_report { overall_health: UNKNOWN } } }
I20260619 08:58:36.272565 15151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:36.270551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40960 (local address 127.0.98.2:43053)
0619 08:58:36.270593 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:36.270595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:36.270606 (+ 11us) server_negotiation.cc:407] Connection header received
0619 08:58:36.270650 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:36.270651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:36.270667 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:36.270718 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:58:36.270808 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:36.271265 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:36.271924 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:36.272164 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:36.272214 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:36.272327 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:58:36.272418 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:58:36.272479 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:36.272492 (+ 13us) server_negotiation.cc:299] Negotiation successful
0619 08:58:36.272510 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 08:58:36.274155 15097 raft_consensus.cc:1261] T 7349478f688042e3b70a0a23aaf3e8ce P ba1b76511e9a401fad69d6e7175118cb [term 1 FOLLOWER]: Refusing update from remote peer b0bb3b26f9e54bc8b16d05572a939d7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:58:36.274174 14827 raft_consensus.cc:1261] T 7349478f688042e3b70a0a23aaf3e8ce P 1d9e834a6cb74090b30ff7e5b108e954 [term 1 FOLLOWER]: Refusing update from remote peer b0bb3b26f9e54bc8b16d05572a939d7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:58:36.274468 15160 consensus_queue.cc:1035] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d9e834a6cb74090b30ff7e5b108e954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:58:36.274592 15157 consensus_queue.cc:1035] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba1b76511e9a401fad69d6e7175118cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:58:36.276986 15163 mvcc.cc:205] Tried to move back new op lower bound from 7298496578656518144 to 7298496578607624192. Current Snapshot: MvccSnapshot[applied={T|T < 7298496578656518144 or (T in {7298496578656518144})}]
I20260619 08:58:36.278723 14685 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:47900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:58:36.278816 14685 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:47900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:58:36.279449 14685 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593: Sending DeleteTablet for 3 replicas of tablet 7349478f688042e3b70a0a23aaf3e8ce
I20260619 08:58:36.279733 15077 tablet_service.cc:1514] Processing DeleteTablet for tablet 7349478f688042e3b70a0a23aaf3e8ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:58:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33848
I20260619 08:58:36.279773 14945 tablet_service.cc:1514] Processing DeleteTablet for tablet 7349478f688042e3b70a0a23aaf3e8ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:58:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40944
I20260619 08:58:36.279795 14807 tablet_service.cc:1514] Processing DeleteTablet for tablet 7349478f688042e3b70a0a23aaf3e8ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:58:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52684
I20260619 08:58:36.279870 14882 tablet_replica.cc:330] T 7349478f688042e3b70a0a23aaf3e8ce P 1d9e834a6cb74090b30ff7e5b108e954: stopping tablet replica
I20260619 08:58:36.279911 14882 raft_consensus.cc:2229] T 7349478f688042e3b70a0a23aaf3e8ce P 1d9e834a6cb74090b30ff7e5b108e954 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:58:36.279942 15172 tablet_replica.cc:330] T 7349478f688042e3b70a0a23aaf3e8ce P ba1b76511e9a401fad69d6e7175118cb: stopping tablet replica
I20260619 08:58:36.279958 14882 raft_consensus.cc:2258] T 7349478f688042e3b70a0a23aaf3e8ce P 1d9e834a6cb74090b30ff7e5b108e954 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:58:36.280023 15172 raft_consensus.cc:2229] T 7349478f688042e3b70a0a23aaf3e8ce P ba1b76511e9a401fad69d6e7175118cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:58:36.280145 15172 raft_consensus.cc:2258] T 7349478f688042e3b70a0a23aaf3e8ce P ba1b76511e9a401fad69d6e7175118cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:58:36.280153 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 14723
I20260619 08:58:36.280946 15172 ts_tablet_manager.cc:1905] T 7349478f688042e3b70a0a23aaf3e8ce P ba1b76511e9a401fad69d6e7175118cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:58:36.282542 15172 ts_tablet_manager.cc:1918] T 7349478f688042e3b70a0a23aaf3e8ce P ba1b76511e9a401fad69d6e7175118cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:58:36.282615 15172 log.cc:1196] T 7349478f688042e3b70a0a23aaf3e8ce P ba1b76511e9a401fad69d6e7175118cb: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/wal/wals/7349478f688042e3b70a0a23aaf3e8ce
I20260619 08:58:36.282896 15172 ts_tablet_manager.cc:1939] T 7349478f688042e3b70a0a23aaf3e8ce P ba1b76511e9a401fad69d6e7175118cb: Deleting consensus metadata
I20260619 08:58:36.283295 14672 catalog_manager.cc:4841] TS ba1b76511e9a401fad69d6e7175118cb (127.0.98.3:33255): tablet 7349478f688042e3b70a0a23aaf3e8ce (table test-table [id=0aa7a79a390f4b0aa7a0645f62a294e9]) successfully deleted
W20260619 08:58:36.285197 14670 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44613 (error 108)
I20260619 08:58:36.285408 15173 tablet_replica.cc:330] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d: stopping tablet replica
I20260619 08:58:36.285403 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 14879
I20260619 08:58:36.285516 15173 raft_consensus.cc:2229] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:58:36.285743 15173 raft_consensus.cc:2258] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:58:36.286257 15149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:36.286132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44613 (local address 127.0.0.1:52696)
0619 08:58:36.286188 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:36.286225 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 08:58:36.286369 14670 catalog_manager.cc:4584] TS 1d9e834a6cb74090b30ff7e5b108e954 (127.0.98.1:44613): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7349478f688042e3b70a0a23aaf3e8ce: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44613: connect: Connection refused (error 111)
I20260619 08:58:36.290112 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 15014
I20260619 08:58:36.290196 15149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:36.290078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43053 (local address 127.0.0.1:40974)
0619 08:58:36.290129 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:36.290163 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 08:58:36.293736 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 14652
I20260619 08:58:36.297986 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46369
--webserver_interface=127.0.98.62
--webserver_port=34429
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46369
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:58:36.376052 15175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:58:36.376288 15175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:58:36.376318 15175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:58:36.377452 15175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:58:36.377490 15175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:58:36.377508 15175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:58:36.377524 15175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:58:36.377539 15175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:58:36.378969 15175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:58:36.379207 15175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:58:36.379410 15175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:58:36.383905 15183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:36.384001 15181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:36.383991 15180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:36.384128 15175 server_base.cc:1034] running on GCE node
I20260619 08:58:36.384356 15175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:58:36.384624 15175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:58:36.385771 15175 hybrid_clock.cc:648] HybridClock initialized: now 1781859516385756 us; error 30 us; skew 500 ppm
Jun 19 08:58:36 dist-test-slave-h4g5 krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:58:36.388026 15175 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 08:58:36.388447 15175 webserver.cc:460] Webserver started at http://127.0.98.62:34429/ using document root <none> and password file <none>
I20260619 08:58:36.388638 15175 fs_manager.cc:359] Metadata directory not provided
I20260619 08:58:36.388684 15175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:58:36.389922 15175 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:58:36.390529 15190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:36.390676 15175 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:36.390753 15175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "8991bbb9aea44da8a3e5b238d53a7593"
format_stamp: "Formatted at 2026-06-19 08:58:34 on dist-test-slave-h4g5"
server_key: "5a56c1513c720f332d576105d2e023d3"
server_key_iv: "3de4fa3ad67e8e0a96dab081147f7383"
server_key_version: "encryptionkey@0"
I20260619 08:58:36.391093 15175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:58:36.402036 15175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:58:36.403683 15175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:58:36.403811 15175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:58:36.407553 15175 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46369
I20260619 08:58:36.407581 15242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46369 every 8 connection(s)
I20260619 08:58:36.407915 15175 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:58:36.409387 15243 sys_catalog.cc:263] Verifying existing consensus state
I20260619 08:58:36.410010 15243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593: Bootstrap starting.
I20260619 08:58:36.412263 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 15175
I20260619 08:58:36.412508 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44613
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46369
--webserver_port=45279
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 08:58:36.417778 15243 log.cc:824] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593: Log is configured to *not* fsync() on all Append() calls
I20260619 08:58:36.419795 15243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 08:58:36.420056 15243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593: Bootstrap complete.
I20260619 08:58:36.422073 15243 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8991bbb9aea44da8a3e5b238d53a7593" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46369 } }
I20260619 08:58:36.422300 15243 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8991bbb9aea44da8a3e5b238d53a7593, State: Initialized, Role: FOLLOWER
I20260619 08:58:36.422439 15243 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8991bbb9aea44da8a3e5b238d53a7593" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46369 } }
I20260619 08:58:36.422503 15243 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:58:36.422544 15243 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:58:36.422590 15243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [term 2 FOLLOWER]: Advancing to term 3
I20260619 08:58:36.423507 15243 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8991bbb9aea44da8a3e5b238d53a7593" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46369 } }
I20260619 08:58:36.423599 15243 leader_election.cc:302] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8991bbb9aea44da8a3e5b238d53a7593; no voters:
I20260619 08:58:36.423746 15243 leader_election.cc:288] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 08:58:36.423789 15247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 08:58:36.423907 15247 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [term 3 LEADER]: Becoming Leader. State: Replica: 8991bbb9aea44da8a3e5b238d53a7593, State: Running, Role: LEADER
I20260619 08:58:36.423997 15243 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:58:36.423986 15247 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8991bbb9aea44da8a3e5b238d53a7593" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46369 } }
I20260619 08:58:36.424279 15248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8991bbb9aea44da8a3e5b238d53a7593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8991bbb9aea44da8a3e5b238d53a7593" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46369 } } }
I20260619 08:58:36.424369 15248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [sys.catalog]: This master's current role is: LEADER
I20260619 08:58:36.424283 15249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8991bbb9aea44da8a3e5b238d53a7593. Latest consensus state: current_term: 3 leader_uuid: "8991bbb9aea44da8a3e5b238d53a7593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8991bbb9aea44da8a3e5b238d53a7593" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46369 } } }
I20260619 08:58:36.424666 15249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8991bbb9aea44da8a3e5b238d53a7593 [sys.catalog]: This master's current role is: LEADER
I20260619 08:58:36.424835 15252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:58:36.425616 15252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:58:36.425717 15252 catalog_manager.cc:1259] Loaded cluster ID: e8c4656838104c6894f1c2d9db3427b1
I20260619 08:58:36.425750 15252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 08:58:36.426005 15252 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f3b9ee329ad google::LogMessage::Fail() at ??:0
@ 0x7f3b9ee3588a google::LogMessage::SendToLog() at ??:0
@ 0x7f3b9ee324de google::LogMessage::Flush() at ??:0
@ 0x7f3b9ee33a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3ba6d530f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f3ba6d5c8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f3ba0d000d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3ba0cf5954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3ba1b596db start_thread at ??:0
@ 0x7f3b9d93061f clone at ??:0
W20260619 08:58:36.523351 15245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:58:36.523541 15245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:58:36.523563 15245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:58:36.524845 15245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:58:36.524887 15245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:58:36.524912 15245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 08:58:36.526328 15245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44613
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45279
--tserver_master_addrs=127.0.98.62:46369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:58:36.526535 15245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:58:36.526754 15245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:58:36.532480 15279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:36.532516 15245 server_base.cc:1034] running on GCE node
W20260619 08:58:36.532546 15282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:36.532469 15280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:36.532837 15245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:58:36.533085 15245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:58:36.534235 15245 hybrid_clock.cc:648] HybridClock initialized: now 1781859516534222 us; error 30 us; skew 500 ppm
Jun 19 08:58:36 dist-test-slave-h4g5 krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:58:36.536557 15245 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 08:58:36.536985 15245 webserver.cc:460] Webserver started at http://127.0.98.1:45279/ using document root <none> and password file <none>
I20260619 08:58:36.537173 15245 fs_manager.cc:359] Metadata directory not provided
I20260619 08:58:36.537211 15245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:58:36.538511 15245 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:58:36.539108 15289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:36.539263 15245 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:36.539342 15245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/wal
uuid: "1d9e834a6cb74090b30ff7e5b108e954"
format_stamp: "Formatted at 2026-06-19 08:58:35 on dist-test-slave-h4g5"
server_key: "a2290833a50be30419b4b21f5df4a2f9"
server_key_iv: "5c02e885c7370220c31eb5030831c96d"
server_key_version: "encryptionkey@0"
I20260619 08:58:36.539677 15245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:58:36.559681 15245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:58:36.562143 15245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:58:36.562301 15245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:58:36.562570 15245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:58:36.563047 15296 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 08:58:36.564566 15245 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 08:58:36.564626 15245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 08:58:36.564661 15245 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 08:58:36.565517 15245 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 08:58:36.565562 15245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 08:58:36.565644 15296 tablet_bootstrap.cc:492] T 7349478f688042e3b70a0a23aaf3e8ce P 1d9e834a6cb74090b30ff7e5b108e954: Bootstrap starting.
I20260619 08:58:36.572664 15245 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44613
I20260619 08:58:36.572850 15403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44613 every 8 connection(s)
I20260619 08:58:36.573127 15245 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
I20260619 08:58:36.575127 15296 log.cc:824] T 7349478f688042e3b70a0a23aaf3e8ce P 1d9e834a6cb74090b30ff7e5b108e954: Log is configured to *not* fsync() on all Append() calls
I20260619 08:58:36.577100 15296 tablet_bootstrap.cc:492] T 7349478f688042e3b70a0a23aaf3e8ce P 1d9e834a6cb74090b30ff7e5b108e954: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 08:58:36.577353 15296 tablet_bootstrap.cc:492] T 7349478f688042e3b70a0a23aaf3e8ce P 1d9e834a6cb74090b30ff7e5b108e954: Bootstrap complete.
I20260619 08:58:36.577389 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 15245
I20260619 08:58:36.577659 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43053
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46369
--webserver_port=34677
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 08:58:36.578150 15296 ts_tablet_manager.cc:1397] T 7349478f688042e3b70a0a23aaf3e8ce P 1d9e834a6cb74090b30ff7e5b108e954: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.008s
I20260619 08:58:36.579217 15296 raft_consensus.cc:348] T 7349478f688042e3b70a0a23aaf3e8ce P 1d9e834a6cb74090b30ff7e5b108e954 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1b76511e9a401fad69d6e7175118cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } } peers { permanent_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43053 } } peers { permanent_uuid: "1d9e834a6cb74090b30ff7e5b108e954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44613 } }
I20260619 08:58:36.579527 15296 raft_consensus.cc:729] T 7349478f688042e3b70a0a23aaf3e8ce P 1d9e834a6cb74090b30ff7e5b108e954 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d9e834a6cb74090b30ff7e5b108e954, State: Initialized, Role: FOLLOWER
I20260619 08:58:36.579677 15296 consensus_queue.cc:260] T 7349478f688042e3b70a0a23aaf3e8ce P 1d9e834a6cb74090b30ff7e5b108e954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1b76511e9a401fad69d6e7175118cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } } peers { permanent_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43053 } } peers { permanent_uuid: "1d9e834a6cb74090b30ff7e5b108e954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44613 } }
I20260619 08:58:36.579933 15296 ts_tablet_manager.cc:1428] T 7349478f688042e3b70a0a23aaf3e8ce P 1d9e834a6cb74090b30ff7e5b108e954: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
W20260619 08:58:36.636931 15406 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:58:36.573857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.1:37067)
0619 08:58:36.574070 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:36.574131 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:36.574804 (+ 673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:36.636760 (+ 61956us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":102,"threads_started":1}
W20260619 08:58:36.637187 15404 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46369 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46369: Client connection negotiation failed: client connection to 127.0.98.62:46369: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 08:58:36.637486 15406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:36.637388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.1:40869)
0619 08:58:36.637432 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:36.637467 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 08:58:36.637789 15406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:36.637705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.1:58517)
0619 08:58:36.637744 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:36.637767 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 08:58:36.637898 15404 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 08:58:36.657999 15408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:58:36.658195 15408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:58:36.658215 15408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:58:36.659385 15408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:58:36.659427 15408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:58:36.659451 15408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 08:58:36.660893 15408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34677
--tserver_master_addrs=127.0.98.62:46369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:58:36.661105 15408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:58:36.661303 15408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:58:36.666178 15415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:36.666185 15414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:36.666213 15408 server_base.cc:1034] running on GCE node
W20260619 08:58:36.666185 15417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:36.666594 15408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:58:36.666808 15408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:58:36.667939 15408 hybrid_clock.cc:648] HybridClock initialized: now 1781859516667925 us; error 29 us; skew 500 ppm
Jun 19 08:58:36 dist-test-slave-h4g5 krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:58:36.670298 15408 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 08:58:36.670674 15408 webserver.cc:460] Webserver started at http://127.0.98.2:34677/ using document root <none> and password file <none>
I20260619 08:58:36.670838 15408 fs_manager.cc:359] Metadata directory not provided
I20260619 08:58:36.670876 15408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:58:36.672125 15408 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:58:36.672677 15424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:36.672808 15408 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:36.672886 15408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/wal
uuid: "b0bb3b26f9e54bc8b16d05572a939d7d"
format_stamp: "Formatted at 2026-06-19 08:58:35 on dist-test-slave-h4g5"
server_key: "caf54e9d0162b428c8b0e0f052afde4d"
server_key_iv: "e2af6d8a5ebb5c02ff8bc957fce5b335"
server_key_version: "encryptionkey@0"
I20260619 08:58:36.673223 15408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:58:36.682286 15408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:58:36.684382 15408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:58:36.684510 15408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:58:36.684710 15408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:58:36.685173 15431 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 08:58:36.686520 15408 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 08:58:36.686578 15408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 08:58:36.686610 15408 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 08:58:36.687243 15408 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 08:58:36.687274 15408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 08:58:36.687345 15431 tablet_bootstrap.cc:492] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d: Bootstrap starting.
I20260619 08:58:36.692761 15408 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43053
I20260619 08:58:36.692977 15538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43053 every 8 connection(s)
I20260619 08:58:36.693138 15408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
I20260619 08:58:36.694139 15541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:36.693777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.2:41933)
0619 08:58:36.693933 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:36.694010 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
W20260619 08:58:36.694339 15539 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46369 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46369: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
I20260619 08:58:36.694592 15541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:36.694505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.2:46625)
0619 08:58:36.694546 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:36.694570 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260619 08:58:36.694823 15541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:36.694752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.2:57693)
0619 08:58:36.694785 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:36.694805 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260619 08:58:36.694908 15539 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 08:58:36.696858 15431 log.cc:824] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d: Log is configured to *not* fsync() on all Append() calls
I20260619 08:58:36.698853 15431 tablet_bootstrap.cc:492] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 08:58:36.699097 15431 tablet_bootstrap.cc:492] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d: Bootstrap complete.
I20260619 08:58:36.699841 15431 ts_tablet_manager.cc:1397] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.007s
I20260619 08:58:36.701052 15431 raft_consensus.cc:348] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1b76511e9a401fad69d6e7175118cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } } peers { permanent_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43053 } } peers { permanent_uuid: "1d9e834a6cb74090b30ff7e5b108e954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44613 } }
I20260619 08:58:36.701309 15431 raft_consensus.cc:729] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0bb3b26f9e54bc8b16d05572a939d7d, State: Initialized, Role: FOLLOWER
I20260619 08:58:36.701435 15431 consensus_queue.cc:260] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1b76511e9a401fad69d6e7175118cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } } peers { permanent_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43053 } } peers { permanent_uuid: "1d9e834a6cb74090b30ff7e5b108e954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44613 } }
I20260619 08:58:36.701673 15431 ts_tablet_manager.cc:1428] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260619 08:58:36.701952 15541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:36.701866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.2:35935)
0619 08:58:36.701904 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:36.701931 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260619 08:58:36.702236 15541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:36.702166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.2:59765)
0619 08:58:36.702202 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:36.702222 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260619 08:58:36.702957 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 15408
I20260619 08:58:36.703167 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33255
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46369
--webserver_port=32985
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:58:36.781185 15544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:58:36.781394 15544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:58:36.781424 15544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:58:36.782575 15544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:58:36.782615 15544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:58:36.782648 15544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 08:58:36.784075 15544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=32985
--tserver_master_addrs=127.0.98.62:46369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:58:36.784348 15544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:58:36.784547 15544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:58:36.789166 15552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:36.789209 15544 server_base.cc:1034] running on GCE node
W20260619 08:58:36.789197 15550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:36.789172 15549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:36.789541 15544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:58:36.789769 15544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:58:36.790920 15544 hybrid_clock.cc:648] HybridClock initialized: now 1781859516790903 us; error 29 us; skew 500 ppm
Jun 19 08:58:36 dist-test-slave-h4g5 krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:58:36.793341 15544 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 08:58:36.793815 15544 webserver.cc:460] Webserver started at http://127.0.98.3:32985/ using document root <none> and password file <none>
I20260619 08:58:36.794003 15544 fs_manager.cc:359] Metadata directory not provided
I20260619 08:58:36.794051 15544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:58:36.795342 15544 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:58:36.795950 15559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:36.796154 15544 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:58:36.796232 15544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/wal
uuid: "ba1b76511e9a401fad69d6e7175118cb"
format_stamp: "Formatted at 2026-06-19 08:58:35 on dist-test-slave-h4g5"
server_key: "24e860de45ea5d1e84425175fae14ea6"
server_key_iv: "03c19c7894473b74e1e493c956ba66cb"
server_key_version: "encryptionkey@0"
I20260619 08:58:36.796634 15544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:58:36.823192 15544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:58:36.825120 15544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:58:36.825254 15544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:58:36.825471 15544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:58:36.825754 15544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:58:36.825788 15544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:36.825809 15544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:58:36.825830 15544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:36.831128 15544 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33255
I20260619 08:58:36.831183 15672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33255 every 8 connection(s)
I20260619 08:58:36.831492 15544 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
I20260619 08:58:36.832435 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:36.832077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.3:51865)
0619 08:58:36.832233 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:36.832299 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":67,"threads_started":1}
W20260619 08:58:36.832798 15673 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46369 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46369: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
I20260619 08:58:36.833025 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:36.832951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.3:33391)
0619 08:58:36.832987 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:36.833010 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260619 08:58:36.833215 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:36.833168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.3:44189)
0619 08:58:36.833187 (+ 19us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:36.833204 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260619 08:58:36.833294 15673 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 08:58:36.837879 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 15544
I20260619 08:58:37.638458 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:37.638241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.1:35973)
0619 08:58:37.638358 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:37.638395 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260619 08:58:37.702867 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:37.702624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.2:55329)
0619 08:58:37.702766 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:37.702810 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":62,"threads_started":1}
I20260619 08:58:37.833794 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:37.833570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.3:59475)
0619 08:58:37.833710 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:37.833744 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":56,"threads_started":1}
I20260619 08:58:38.337352 15680 raft_consensus.cc:482] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:58:38.337493 15680 raft_consensus.cc:504] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1b76511e9a401fad69d6e7175118cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } } peers { permanent_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43053 } } peers { permanent_uuid: "1d9e834a6cb74090b30ff7e5b108e954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44613 } }
I20260619 08:58:38.337852 15680 leader_election.cc:288] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ba1b76511e9a401fad69d6e7175118cb (127.0.98.3:33255), 1d9e834a6cb74090b30ff7e5b108e954 (127.0.98.1:44613)
Jun 19 08:58:38 dist-test-slave-h4g5 krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
Jun 19 08:58:38 dist-test-slave-h4g5 krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260619 08:58:38.347752 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:38.338026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44613 (local address 127.0.98.2:58255)
0619 08:58:38.338154 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:38.338182 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:38.338374 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:38.338722 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:38.338725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:38.338994 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:58:38.339184 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:38.339190 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:38.340270 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:38.340271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:38.341099 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:38.341102 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:38.341647 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:38.341648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:38.341746 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:38.342272 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:58:38.342280 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:58:38.343792 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:58:38.345872 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:38.345877 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:38.345879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:38.346199 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:38.346476 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:38.346477 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:38.346477 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:38.346636 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:38.347202 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:58:38.347203 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:58:38.347347 (+ 144us) client_negotiation.cc:769] Sending connection context
0619 08:58:38.347460 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 08:58:38.347640 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":34,"thread_start_us":59,"threads_started":1}
I20260619 08:58:38.347776 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:38.338162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48993 (local address 127.0.98.3:33255)
0619 08:58:38.338331 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:38.338341 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:38.338369 (+ 28us) server_negotiation.cc:407] Connection header received
0619 08:58:38.338952 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:38.338954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:38.338991 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:38.339097 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:38.339402 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:38.340276 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:38.341174 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:38.341563 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:38.341656 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:38.344304 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:38.344308 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:38.344308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:38.344317 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:38.345895 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:38.346280 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:38.346283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:38.346284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:38.346409 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:38.346748 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:38.346749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:38.346749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:38.347102 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:38.347192 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:38.347533 (+ 341us) server_negotiation.cc:299] Negotiation successful
0619 08:58:38.347670 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":57,"threads_started":1}
I20260619 08:58:38.347808 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:38.338026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33255 (local address 127.0.98.2:48993)
0619 08:58:38.338156 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:38.338182 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:38.338374 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:38.339098 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:38.339099 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:38.339257 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:58:38.339317 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:38.339319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:38.340355 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:38.340356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:38.341100 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:38.341102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:38.341637 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:38.341638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:38.341745 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:38.342338 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:58:38.342343 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:58:38.344218 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:58:38.345963 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:38.345963 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:38.345964 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:38.346199 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:38.346476 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:38.346477 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:38.346477 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:38.346636 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:38.347185 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:58:38.347192 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:58:38.347347 (+ 155us) client_negotiation.cc:769] Sending connection context
0619 08:58:38.347460 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 08:58:38.347702 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":154,"thread_start_us":44,"threads_started":1}
I20260619 08:58:38.347792 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:38.338131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58255 (local address 127.0.98.1:44613)
0619 08:58:38.338444 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:38.338448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:38.338458 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:58:38.338549 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:38.338551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:38.338574 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:38.338651 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:38.339291 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:38.340192 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:38.341174 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:38.341576 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:38.341680 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:38.343900 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:38.343904 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:38.343904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:38.343914 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:38.345794 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:38.346280 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:38.346283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:38.346284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:38.346409 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:38.346755 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:38.346756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:38.346756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:38.347141 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:38.347215 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:38.347533 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 08:58:38.347705 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":73,"threads_started":1}
I20260619 08:58:38.348454 15358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7349478f688042e3b70a0a23aaf3e8ce" candidate_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1d9e834a6cb74090b30ff7e5b108e954" is_pre_election: true
I20260619 08:58:38.348619 15358 raft_consensus.cc:2454] T 7349478f688042e3b70a0a23aaf3e8ce P 1d9e834a6cb74090b30ff7e5b108e954 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0bb3b26f9e54bc8b16d05572a939d7d in term 1.
W20260619 08:58:38.348641 15428 leader_election.cc:341] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ba1b76511e9a401fad69d6e7175118cb (127.0.98.3:33255): Not found: Tablet not found: 7349478f688042e3b70a0a23aaf3e8ce
I20260619 08:58:38.348889 15426 leader_election.cc:302] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d9e834a6cb74090b30ff7e5b108e954, b0bb3b26f9e54bc8b16d05572a939d7d; no voters: ba1b76511e9a401fad69d6e7175118cb
I20260619 08:58:38.349005 15680 raft_consensus.cc:2781] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 08:58:38.349068 15680 raft_consensus.cc:482] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:58:38.349092 15680 raft_consensus.cc:3037] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [term 1 FOLLOWER]: Advancing to term 2
I20260619 08:58:38.350203 15680 raft_consensus.cc:504] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1b76511e9a401fad69d6e7175118cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } } peers { permanent_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43053 } } peers { permanent_uuid: "1d9e834a6cb74090b30ff7e5b108e954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44613 } }
I20260619 08:58:38.350328 15680 leader_election.cc:288] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [CANDIDATE]: Term 2 election: Requested vote from peers ba1b76511e9a401fad69d6e7175118cb (127.0.98.3:33255), 1d9e834a6cb74090b30ff7e5b108e954 (127.0.98.1:44613)
I20260619 08:58:38.350489 15358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7349478f688042e3b70a0a23aaf3e8ce" candidate_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1d9e834a6cb74090b30ff7e5b108e954"
I20260619 08:58:38.350575 15358 raft_consensus.cc:3037] T 7349478f688042e3b70a0a23aaf3e8ce P 1d9e834a6cb74090b30ff7e5b108e954 [term 1 FOLLOWER]: Advancing to term 2
W20260619 08:58:38.350602 15428 leader_election.cc:341] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer ba1b76511e9a401fad69d6e7175118cb (127.0.98.3:33255): Not found: Tablet not found: 7349478f688042e3b70a0a23aaf3e8ce
I20260619 08:58:38.351657 15358 raft_consensus.cc:2454] T 7349478f688042e3b70a0a23aaf3e8ce P 1d9e834a6cb74090b30ff7e5b108e954 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0bb3b26f9e54bc8b16d05572a939d7d in term 2.
I20260619 08:58:38.351822 15426 leader_election.cc:302] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d9e834a6cb74090b30ff7e5b108e954, b0bb3b26f9e54bc8b16d05572a939d7d; no voters: ba1b76511e9a401fad69d6e7175118cb
I20260619 08:58:38.351919 15680 raft_consensus.cc:2781] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [term 2 FOLLOWER]: Leader election won for term 2
I20260619 08:58:38.352020 15680 raft_consensus.cc:686] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [term 2 LEADER]: Becoming Leader. State: Replica: b0bb3b26f9e54bc8b16d05572a939d7d, State: Running, Role: LEADER
I20260619 08:58:38.352118 15680 consensus_queue.cc:237] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1b76511e9a401fad69d6e7175118cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } } peers { permanent_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43053 } } peers { permanent_uuid: "1d9e834a6cb74090b30ff7e5b108e954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44613 } }
I20260619 08:58:38.352317 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:38.352229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.2:50175)
0619 08:58:38.352258 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:38.352286 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260619 08:58:38.639179 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:38.638933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.1:50713)
0619 08:58:38.639091 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:38.639133 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":74,"threads_started":1}
I20260619 08:58:38.834447 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:38.834208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.3:45063)
0619 08:58:38.834351 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:38.834398 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
I20260619 08:58:38.934631 15358 raft_consensus.cc:1261] T 7349478f688042e3b70a0a23aaf3e8ce P 1d9e834a6cb74090b30ff7e5b108e954 [term 2 FOLLOWER]: Refusing update from remote peer b0bb3b26f9e54bc8b16d05572a939d7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260619 08:58:38.934973 15689 consensus_queue.cc:1035] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d9e834a6cb74090b30ff7e5b108e954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260619 08:58:38.935799 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:38.935669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.1:54801)
0619 08:58:38.935724 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:38.935766 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 08:58:38.936707 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:38.936552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.2:49311)
0619 08:58:38.936634 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:38.936668 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68,"thread_start_us":45,"threads_started":1}
I20260619 08:58:38.936798 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:38.936713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.1:59511)
0619 08:58:38.936749 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:38.936774 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260619 08:58:38.936775 15428 consensus_peers.cc:487] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d -> Peer ba1b76511e9a401fad69d6e7175118cb (127.0.98.3:33255): Couldn't send request to peer ba1b76511e9a401fad69d6e7175118cb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7349478f688042e3b70a0a23aaf3e8ce. This is attempt 1: this message will repeat every 5th retry.
I20260619 08:58:39.455034 15697 ts_tablet_manager.cc:927] T 7349478f688042e3b70a0a23aaf3e8ce P ba1b76511e9a401fad69d6e7175118cb: Initiating tablet copy from peer b0bb3b26f9e54bc8b16d05572a939d7d (127.0.98.2:43053)
I20260619 08:58:39.455229 15697 tablet_copy_client.cc:316] T 7349478f688042e3b70a0a23aaf3e8ce P ba1b76511e9a401fad69d6e7175118cb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.2:43053
Jun 19 08:58:39 dist-test-slave-h4g5 krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260619 08:58:39.462685 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:39.455572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53401 (local address 127.0.98.2:43053)
0619 08:58:39.455674 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:39.455679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:39.455690 (+ 11us) server_negotiation.cc:407] Connection header received
0619 08:58:39.455753 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:39.455755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:39.455767 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:39.455831 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:39.456335 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:39.457034 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:39.457786 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:39.458106 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:39.458161 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:39.460061 (+ 1900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:39.460070 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:39.460071 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:39.460113 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:39.461632 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:39.462006 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:39.462008 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:39.462009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:39.462064 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:39.462211 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:39.462211 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:39.462212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:39.462368 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:39.462430 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:39.462550 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 08:58:39.462615 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":38,"threads_started":1}
I20260619 08:58:39.462687 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:39.455489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43053 (local address 127.0.98.3:53401)
0619 08:58:39.455638 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:39.455653 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:39.455709 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:39.455839 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:39.455841 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:39.456200 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:58:39.456273 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:39.456276 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:39.457111 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:39.457112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:39.457718 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:39.457721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:39.458163 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:39.458164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:39.458222 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:39.458639 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:58:39.458643 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:58:39.459988 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:58:39.461710 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:39.461717 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:39.461719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:39.461937 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:39.462115 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:39.462115 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:39.462116 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:39.462169 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:39.462417 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:58:39.462422 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:58:39.462466 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 08:58:39.462522 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 08:58:39.462613 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":74,"threads_started":1}
I20260619 08:58:39.463135 15513 tablet_copy_service.cc:140] P b0bb3b26f9e54bc8b16d05572a939d7d: Received BeginTabletCopySession request for tablet 7349478f688042e3b70a0a23aaf3e8ce from peer ba1b76511e9a401fad69d6e7175118cb ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53401)
I20260619 08:58:39.463203 15513 tablet_copy_service.cc:161] P b0bb3b26f9e54bc8b16d05572a939d7d: Beginning new tablet copy session on tablet 7349478f688042e3b70a0a23aaf3e8ce from peer ba1b76511e9a401fad69d6e7175118cb at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53401: session id = ba1b76511e9a401fad69d6e7175118cb-7349478f688042e3b70a0a23aaf3e8ce
I20260619 08:58:39.463665 15513 tablet_copy_source_session.cc:205] T 7349478f688042e3b70a0a23aaf3e8ce P b0bb3b26f9e54bc8b16d05572a939d7d: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 08:58:39.464676 15697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7349478f688042e3b70a0a23aaf3e8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:58:39.467233 15697 tablet_copy_client.cc:791] T 7349478f688042e3b70a0a23aaf3e8ce P ba1b76511e9a401fad69d6e7175118cb: tablet copy: Starting download of 0 data blocks...
I20260619 08:58:39.467344 15697 tablet_copy_client.cc:655] T 7349478f688042e3b70a0a23aaf3e8ce P ba1b76511e9a401fad69d6e7175118cb: tablet copy: Starting download of 1 WAL segments...
I20260619 08:58:39.468180 15697 tablet_copy_client.cc:531] T 7349478f688042e3b70a0a23aaf3e8ce P ba1b76511e9a401fad69d6e7175118cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 08:58:39.468806 15697 tablet_bootstrap.cc:492] T 7349478f688042e3b70a0a23aaf3e8ce P ba1b76511e9a401fad69d6e7175118cb: Bootstrap starting.
I20260619 08:58:39.470114 15697 log.cc:824] T 7349478f688042e3b70a0a23aaf3e8ce P ba1b76511e9a401fad69d6e7175118cb: Log is configured to *not* fsync() on all Append() calls
I20260619 08:58:39.472442 15697 tablet_bootstrap.cc:492] T 7349478f688042e3b70a0a23aaf3e8ce P ba1b76511e9a401fad69d6e7175118cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 08:58:39.472608 15697 tablet_bootstrap.cc:492] T 7349478f688042e3b70a0a23aaf3e8ce P ba1b76511e9a401fad69d6e7175118cb: Bootstrap complete.
I20260619 08:58:39.472699 15697 ts_tablet_manager.cc:1397] T 7349478f688042e3b70a0a23aaf3e8ce P ba1b76511e9a401fad69d6e7175118cb: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260619 08:58:39.473603 15697 raft_consensus.cc:348] T 7349478f688042e3b70a0a23aaf3e8ce P ba1b76511e9a401fad69d6e7175118cb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1b76511e9a401fad69d6e7175118cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } } peers { permanent_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43053 } } peers { permanent_uuid: "1d9e834a6cb74090b30ff7e5b108e954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44613 } }
I20260619 08:58:39.473737 15697 raft_consensus.cc:729] T 7349478f688042e3b70a0a23aaf3e8ce P ba1b76511e9a401fad69d6e7175118cb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba1b76511e9a401fad69d6e7175118cb, State: Initialized, Role: FOLLOWER
I20260619 08:58:39.473843 15697 consensus_queue.cc:260] T 7349478f688042e3b70a0a23aaf3e8ce P ba1b76511e9a401fad69d6e7175118cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1b76511e9a401fad69d6e7175118cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33255 } } peers { permanent_uuid: "b0bb3b26f9e54bc8b16d05572a939d7d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43053 } } peers { permanent_uuid: "1d9e834a6cb74090b30ff7e5b108e954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44613 } }
I20260619 08:58:39.474093 15697 ts_tablet_manager.cc:1428] T 7349478f688042e3b70a0a23aaf3e8ce P ba1b76511e9a401fad69d6e7175118cb: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 08:58:39.474351 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:39.474263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.3:51517)
0619 08:58:39.474299 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:39.474326 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260619 08:58:39.474411 15513 tablet_copy_service.cc:342] P b0bb3b26f9e54bc8b16d05572a939d7d: Request end of tablet copy session ba1b76511e9a401fad69d6e7175118cb-7349478f688042e3b70a0a23aaf3e8ce received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53401
I20260619 08:58:39.474466 15513 tablet_copy_service.cc:433] P b0bb3b26f9e54bc8b16d05572a939d7d: ending tablet copy session ba1b76511e9a401fad69d6e7175118cb-7349478f688042e3b70a0a23aaf3e8ce on tablet 7349478f688042e3b70a0a23aaf3e8ce with peer ba1b76511e9a401fad69d6e7175118cb
I20260619 08:58:39.474855 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:39.474752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.3:38923)
0619 08:58:39.474798 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:39.474827 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 08:58:39.937373 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:39.937190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.2:52201)
0619 08:58:39.937297 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:39.937331 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":50,"threads_started":1}
I20260619 08:58:39.937373 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:39.937197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.1:46123)
0619 08:58:39.937297 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:39.937329 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":46,"threads_started":1}
I20260619 08:58:39.954391 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:39.954276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.2:56567)
0619 08:58:39.954322 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:39.954353 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 08:58:39.954411 15627 raft_consensus.cc:1203] T 7349478f688042e3b70a0a23aaf3e8ce P ba1b76511e9a401fad69d6e7175118cb [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260619 08:58:39.954845 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:39.954721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.3:40013)
0619 08:58:39.954772 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:39.954814 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 08:58:40.937996 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:40.937790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.1:52277)
0619 08:58:40.937908 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:40.937949 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":67,"threads_started":1}
I20260619 08:58:40.955062 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:40.954824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.2:45663)
0619 08:58:40.954972 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:40.955013 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":65,"threads_started":1}
I20260619 08:58:40.955471 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:40.955277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.3:52433)
0619 08:58:40.955375 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:40.955421 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":49,"threads_started":1}
I20260619 08:58:41.938699 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:41.938463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.1:35643)
0619 08:58:41.938606 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:41.938650 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":62,"threads_started":1}
I20260619 08:58:41.955801 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:41.955550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.2:37473)
0619 08:58:41.955699 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:41.955741 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":62,"threads_started":1}
I20260619 08:58:41.956111 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:41.955894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.3:42183)
0619 08:58:41.956011 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:41.956043 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260619 08:58:41.956157 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:41.956045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.2:60537)
0619 08:58:41.956105 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:41.956133 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 08:58:42.939337 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:42.939104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.1:41469)
0619 08:58:42.939250 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:42.939288 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":65,"threads_started":1}
I20260619 08:58:42.956761 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:42.956542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.3:38595)
0619 08:58:42.956684 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:42.956714 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":72,"threads_started":1}
I20260619 08:58:42.956763 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:42.956540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.2:41937)
0619 08:58:42.956682 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:42.956721 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":70,"threads_started":1}
I20260619 08:58:43.940145 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:43.939825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.1:55615)
0619 08:58:43.940014 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:43.940060 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":82,"threads_started":1}
I20260619 08:58:43.957394 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:43.957186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.3:36439)
0619 08:58:43.957316 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:43.957350 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
I20260619 08:58:43.957415 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:43.957186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.2:41825)
0619 08:58:43.957310 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:43.957345 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
I20260619 08:58:44.940723 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:44.940531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.1:54217)
0619 08:58:44.940644 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:44.940676 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":66,"threads_started":1}
I20260619 08:58:44.958081 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:44.957859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.2:49467)
0619 08:58:44.957993 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:44.958028 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":71,"threads_started":1}
I20260619 08:58:44.958103 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:44.957854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.3:44075)
0619 08:58:44.958017 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:44.958058 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":73,"threads_started":1}
I20260619 08:58:45.941377 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:45.941142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.1:41451)
0619 08:58:45.941286 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:45.941329 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":69,"threads_started":1}
I20260619 08:58:45.958712 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:45.958475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.2:49943)
0619 08:58:45.958615 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:45.958652 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
I20260619 08:58:45.958706 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:45.958522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.3:46809)
0619 08:58:45.958633 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:45.958656 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":54,"threads_started":1}
I20260619 08:58:46.942070 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:46.941850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.1:55585)
0619 08:58:46.941979 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:46.942022 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
I20260619 08:58:46.959411 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:46.959188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.3:58325)
0619 08:58:46.959335 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:46.959368 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":63,"threads_started":1}
I20260619 08:58:46.959414 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:46.959189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.2:32819)
0619 08:58:46.959327 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:46.959365 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":67,"threads_started":1}
I20260619 08:58:47.942731 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:47.942515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.1:54359)
0619 08:58:47.942640 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:47.942685 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":66,"threads_started":1}
I20260619 08:58:47.960071 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:47.959862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.2:48905)
0619 08:58:47.959991 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:47.960022 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":66,"threads_started":1}
I20260619 08:58:47.960071 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:47.959864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.3:60889)
0619 08:58:47.959990 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:47.960022 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260619 08:58:48.943419 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:48.943169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.1:38025)
0619 08:58:48.943327 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:48.943372 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":84,"threads_started":1}
I20260619 08:58:48.960779 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:48.960563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.3:44287)
0619 08:58:48.960695 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:48.960734 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":57,"threads_started":1}
I20260619 08:58:48.960783 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:48.960580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.2:50825)
0619 08:58:48.960706 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:48.960741 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
I20260619 08:58:49.944057 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:49.943798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.1:47357)
0619 08:58:49.943948 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:49.943997 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":59,"threads_started":1}
I20260619 08:58:49.961472 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:49.961254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.3:57347)
0619 08:58:49.961388 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:49.961424 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":63,"threads_started":1}
I20260619 08:58:49.961480 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:49.961254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.2:35873)
0619 08:58:49.961394 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:49.961432 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":72,"threads_started":1}
I20260619 08:58:50.944677 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:50.944478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.1:41867)
0619 08:58:50.944592 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:50.944630 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
I20260619 08:58:50.962100 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:50.961883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.2:36319)
0619 08:58:50.962015 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:50.962051 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
I20260619 08:58:50.962117 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:50.961883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46369 (local address 127.0.98.3:47203)
0619 08:58:50.962028 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:50.962073 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":64,"threads_started":1}
I20260619 08:58:51.845057 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 15245
I20260619 08:58:51.849233 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 15408
I20260619 08:58:51.853541 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 15544
2026-06-19T08:58:51Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18122 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:58:51 dist-test-slave-h4g5 krb5kdc[15741](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:58:51 dist-test-slave-h4g5 krb5kdc[15741](info): set up 2 sockets
Jun 19 08:58:51 dist-test-slave-h4g5 krb5kdc[15741](info): commencing operation
krb5kdc: starting...
W20260619 08:58:52.799886 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 08:58:52 dist-test-slave-h4g5 krb5kdc[15741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:58:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:58:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 08:58:52.912348 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41155
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:58:52.991261 15754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:58:52.991479 15754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:58:52.991501 15754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:58:52.992749 15754 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 08:58:52.992796 15754 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 08:58:52.992808 15754 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 08:58:52.992818 15754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:58:52.992830 15754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:58:52.992841 15754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:58:52.992856 15754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:58:52.992879 15754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:58:52.994412 15754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:58:52.994665 15754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:58:52.994880 15754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:58:52.999733 15759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:52.999732 15754 server_base.cc:1034] running on GCE node
W20260619 08:58:52.999750 15760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:52.999729 15762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:53.000232 15754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:58:53.000522 15754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:58:53.001662 15754 hybrid_clock.cc:648] HybridClock initialized: now 1781859533001650 us; error 33 us; skew 500 ppm
Jun 19 08:58:52 dist-test-slave-h4g5 krb5kdc[15741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:58:53.004141 15754 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 08:58:53.004571 15754 webserver.cc:460] Webserver started at http://127.0.98.62:40485/ using document root <none> and password file <none>
I20260619 08:58:53.004758 15754 fs_manager.cc:359] Metadata directory not provided
I20260619 08:58:53.004796 15754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:58:53.004899 15754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:58:53.005887 15754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "19abdfe959a1427f91d2c92fa822266b"
format_stamp: "Formatted at 2026-06-19 08:58:53 on dist-test-slave-h4g5"
server_key: "39e9d43f289b6845e9ce204af44cd31c"
server_key_iv: "01c7fda7edd719f6f86a5c354ff89914"
server_key_version: "encryptionkey@0"
I20260619 08:58:53.006206 15754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "19abdfe959a1427f91d2c92fa822266b"
format_stamp: "Formatted at 2026-06-19 08:58:53 on dist-test-slave-h4g5"
server_key: "39e9d43f289b6845e9ce204af44cd31c"
server_key_iv: "01c7fda7edd719f6f86a5c354ff89914"
server_key_version: "encryptionkey@0"
I20260619 08:58:53.007402 15754 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 08:58:53.008028 15769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:53.008184 15754 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:58:53.008255 15754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "19abdfe959a1427f91d2c92fa822266b"
format_stamp: "Formatted at 2026-06-19 08:58:53 on dist-test-slave-h4g5"
server_key: "39e9d43f289b6845e9ce204af44cd31c"
server_key_iv: "01c7fda7edd719f6f86a5c354ff89914"
server_key_version: "encryptionkey@0"
I20260619 08:58:53.008339 15754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:58:53.027033 15754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:58:53.028872 15754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:58:53.028986 15754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:58:53.032819 15754 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41155
I20260619 08:58:53.032867 15821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41155 every 8 connection(s)
I20260619 08:58:53.033190 15754 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:58:53.034039 15822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:58:53.036361 15822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b: Bootstrap starting.
I20260619 08:58:53.036623 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 15754
I20260619 08:58:53.036700 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 08:58:53.036870 392 external_mini_cluster.cc:1428] Setting key 13c3fe1502b1426fc3e40a60de66f936
I20260619 08:58:53.036989 15822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b: Neither blocks nor log segments found. Creating new log.
I20260619 08:58:53.037317 15822 log.cc:824] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b: Log is configured to *not* fsync() on all Append() calls
I20260619 08:58:53.037993 15822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b: No bootstrap required, opened a new log
I20260619 08:58:53.039767 15822 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19abdfe959a1427f91d2c92fa822266b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41155 } }
I20260619 08:58:53.039902 15822 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:58:53.039933 15822 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19abdfe959a1427f91d2c92fa822266b, State: Initialized, Role: FOLLOWER
I20260619 08:58:53.040057 15822 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19abdfe959a1427f91d2c92fa822266b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41155 } }
I20260619 08:58:53.040159 15822 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:58:53.040200 15822 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:58:53.040248 15822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:58:53.040755 15822 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19abdfe959a1427f91d2c92fa822266b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41155 } }
I20260619 08:58:53.040844 15822 leader_election.cc:302] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19abdfe959a1427f91d2c92fa822266b; no voters:
I20260619 08:58:53.041009 15822 leader_election.cc:288] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:58:53.041051 15827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:58:53.041193 15827 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [term 1 LEADER]: Becoming Leader. State: Replica: 19abdfe959a1427f91d2c92fa822266b, State: Running, Role: LEADER
I20260619 08:58:53.041285 15822 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:58:53.041334 15827 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19abdfe959a1427f91d2c92fa822266b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41155 } }
I20260619 08:58:53.041698 15827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19abdfe959a1427f91d2c92fa822266b. Latest consensus state: current_term: 1 leader_uuid: "19abdfe959a1427f91d2c92fa822266b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19abdfe959a1427f91d2c92fa822266b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41155 } } }
I20260619 08:58:53.041790 15829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19abdfe959a1427f91d2c92fa822266b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19abdfe959a1427f91d2c92fa822266b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41155 } } }
I20260619 08:58:53.041859 15827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [sys.catalog]: This master's current role is: LEADER
I20260619 08:58:53.041913 15829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [sys.catalog]: This master's current role is: LEADER
I20260619 08:58:53.042161 15836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:58:53.042711 15836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 08:58:53 dist-test-slave-h4g5 krb5kdc[15741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:58:53.044842 15836 catalog_manager.cc:1347] Generated new cluster ID: 702f520165994b689af549353709df7b
I20260619 08:58:53.044929 15836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:58:53.047003 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:53.037329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33664 (local address 127.0.98.62:41155)
0619 08:58:53.037501 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:53.037510 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:53.037542 (+ 32us) server_negotiation.cc:407] Connection header received
0619 08:58:53.038007 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:53.038014 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:53.038062 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:53.038208 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:53.038797 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.039672 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.040581 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.041043 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.041145 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:53.043947 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:53.043951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:53.043951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:53.043964 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:53.045603 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:53.045911 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:53.045914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:53.045915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:53.046008 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:53.046183 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:53.046185 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:53.046185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:53.046412 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:53.046546 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:53.046658 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 08:58:53.046787 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":73,"threads_started":1}
I20260619 08:58:53.069811 15836 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:58:53.070390 15836 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:58:53.081513 15836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b: Generated new TSK 0
I20260619 08:58:53.081748 15836 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 08:58:53.108686 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41155
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:58:53.189136 15848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:58:53.189342 15848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:58:53.189373 15848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:58:53.190512 15848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:58:53.190554 15848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:58:53.190588 15848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 08:58:53.192039 15848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:58:53.192288 15848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:58:53.192499 15848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:58:53.197321 15853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:53.197321 15856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:53.197356 15854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:53.197512 15848 server_base.cc:1034] running on GCE node
I20260619 08:58:53.197810 15848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:58:53.198048 15848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:58:53.199218 15848 hybrid_clock.cc:648] HybridClock initialized: now 1781859533199200 us; error 32 us; skew 500 ppm
Jun 19 08:58:53 dist-test-slave-h4g5 krb5kdc[15741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:58:53.201663 15848 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 08:58:53.202071 15848 webserver.cc:460] Webserver started at http://127.0.98.1:39765/ using document root <none> and password file <none>
I20260619 08:58:53.202263 15848 fs_manager.cc:359] Metadata directory not provided
I20260619 08:58:53.202309 15848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:58:53.202432 15848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:58:53.203454 15848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/data/instance:
uuid: "28eabc22c4e04f059339795b02f88ed5"
format_stamp: "Formatted at 2026-06-19 08:58:53 on dist-test-slave-h4g5"
server_key: "263490f766468d0e26e514905f24c728"
server_key_iv: "a746b61d4dac9f74b82e5c0460297042"
server_key_version: "encryptionkey@0"
I20260619 08:58:53.203799 15848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/wal/instance:
uuid: "28eabc22c4e04f059339795b02f88ed5"
format_stamp: "Formatted at 2026-06-19 08:58:53 on dist-test-slave-h4g5"
server_key: "263490f766468d0e26e514905f24c728"
server_key_iv: "a746b61d4dac9f74b82e5c0460297042"
server_key_version: "encryptionkey@0"
I20260619 08:58:53.205087 15848 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:58:53.205801 15863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:53.205958 15848 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 08:58:53.206051 15848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/wal
uuid: "28eabc22c4e04f059339795b02f88ed5"
format_stamp: "Formatted at 2026-06-19 08:58:53 on dist-test-slave-h4g5"
server_key: "263490f766468d0e26e514905f24c728"
server_key_iv: "a746b61d4dac9f74b82e5c0460297042"
server_key_version: "encryptionkey@0"
I20260619 08:58:53.206118 15848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:58:53.214233 15848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:58:53.215931 15848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:58:53.216046 15848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:58:53.216274 15848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:58:53.216523 15848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:58:53.216552 15848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:53.216571 15848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:58:53.216588 15848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:53.221846 15848 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40207
I20260619 08:58:53.221915 15976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40207 every 8 connection(s)
I20260619 08:58:53.222189 15848 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
I20260619 08:58:53.223505 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 15848
I20260619 08:58:53.223605 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/wal/instance
I20260619 08:58:53.223793 392 external_mini_cluster.cc:1428] Setting key 0c1ebadd4c6ca7240ccf3eba750eed02
Jun 19 08:58:53 dist-test-slave-h4g5 krb5kdc[15741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 08:58:53.231794 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:53.222865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41225 (local address 127.0.98.62:41155)
0619 08:58:53.222921 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:53.222923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:53.223121 (+ 198us) server_negotiation.cc:407] Connection header received
0619 08:58:53.223627 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:53.223628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:53.223641 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:53.223704 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:53.224788 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.225536 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.226508 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.226749 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.226806 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:53.229140 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:53.229141 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:53.229141 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:53.229147 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:53.230380 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:53.230825 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:53.230825 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:53.230825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:53.230870 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:53.231130 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:53.231130 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:53.231130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:53.231224 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:53.231267 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:53.231617 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 08:58:53.231711 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 08:58:53.231971 15979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:53.222828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.1:41225)
0619 08:58:53.222989 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:53.223038 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:53.223520 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:53.223781 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:53.223784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:53.224505 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:58:53.224699 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:53.224704 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.225621 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.225622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:53.226429 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:53.226431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.226812 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.226812 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:53.226904 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:53.227394 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:58:53.227402 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:58:53.229035 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:58:53.230461 (+ 1426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:53.230468 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:53.230470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:53.230754 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:53.230924 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:53.230925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:53.230925 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:53.231069 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:53.231278 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:58:53.231284 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:58:53.231453 (+ 169us) client_negotiation.cc:769] Sending connection context
0619 08:58:53.231570 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 08:58:53.231739 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
I20260619 08:58:53.232419 15977 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41155
I20260619 08:58:53.232507 15977 heartbeater.cc:464] Registering TS with master...
I20260619 08:58:53.232677 15977 heartbeater.cc:511] Master 127.0.98.62:41155 requested a full tablet report, sending...
I20260619 08:58:53.233120 15786 ts_manager.cc:194] Registered new tserver with Master: 28eabc22c4e04f059339795b02f88ed5 (127.0.98.1:40207)
I20260619 08:58:53.233771 15786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41225
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 08:58:53.247530 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41155
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:58:53.328377 15982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:58:53.328619 15982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:58:53.328655 15982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:58:53.329854 15982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:58:53.329903 15982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:58:53.329941 15982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 08:58:53.331444 15982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:58:53.331691 15982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:58:53.331907 15982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:58:53.336916 15987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:53.336918 15990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:53.336917 15988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:53.337067 15982 server_base.cc:1034] running on GCE node
I20260619 08:58:53.337332 15982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:58:53.337534 15982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:58:53.338685 15982 hybrid_clock.cc:648] HybridClock initialized: now 1781859533338668 us; error 29 us; skew 500 ppm
Jun 19 08:58:53 dist-test-slave-h4g5 krb5kdc[15741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:58:53.341120 15982 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 08:58:53.341614 15982 webserver.cc:460] Webserver started at http://127.0.98.2:46081/ using document root <none> and password file <none>
I20260619 08:58:53.341814 15982 fs_manager.cc:359] Metadata directory not provided
I20260619 08:58:53.341884 15982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:58:53.342005 15982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:58:53.343068 15982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/data/instance:
uuid: "decab1e0f88a49b9922817b18178c8d2"
format_stamp: "Formatted at 2026-06-19 08:58:53 on dist-test-slave-h4g5"
server_key: "77d3931545129c85429db132f12e298a"
server_key_iv: "b145e7877e8ffc94e458473bcdc12c75"
server_key_version: "encryptionkey@0"
I20260619 08:58:53.343390 15982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/wal/instance:
uuid: "decab1e0f88a49b9922817b18178c8d2"
format_stamp: "Formatted at 2026-06-19 08:58:53 on dist-test-slave-h4g5"
server_key: "77d3931545129c85429db132f12e298a"
server_key_iv: "b145e7877e8ffc94e458473bcdc12c75"
server_key_version: "encryptionkey@0"
I20260619 08:58:53.344669 15982 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:58:53.345350 15997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:53.345487 15982 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:58:53.345574 15982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/wal
uuid: "decab1e0f88a49b9922817b18178c8d2"
format_stamp: "Formatted at 2026-06-19 08:58:53 on dist-test-slave-h4g5"
server_key: "77d3931545129c85429db132f12e298a"
server_key_iv: "b145e7877e8ffc94e458473bcdc12c75"
server_key_version: "encryptionkey@0"
I20260619 08:58:53.345633 15982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:58:53.359524 15982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:58:53.361733 15982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:58:53.361881 15982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:58:53.362088 15982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:58:53.362361 15982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:58:53.362392 15982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:53.362421 15982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:58:53.362444 15982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:53.368227 15982 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45001
I20260619 08:58:53.368283 16110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45001 every 8 connection(s)
I20260619 08:58:53.368570 15982 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
I20260619 08:58:53.371984 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 15982
I20260619 08:58:53.372071 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/wal/instance
I20260619 08:58:53.372291 392 external_mini_cluster.cc:1428] Setting key 5df9b93f6f38b6af68b79b18db0403a0
Jun 19 08:58:53 dist-test-slave-h4g5 krb5kdc[15741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:58:53.378079 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:53.369218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57123 (local address 127.0.98.62:41155)
0619 08:58:53.369278 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:53.369280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:53.369462 (+ 182us) server_negotiation.cc:407] Connection header received
0619 08:58:53.369974 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:53.369975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:53.369989 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:53.370039 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:53.370996 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.371491 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.372635 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.372904 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.372972 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:53.375359 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:53.375360 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:53.375361 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:53.375368 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:53.376643 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:53.377163 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:53.377164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:53.377164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:53.377215 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:53.377474 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:53.377475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:53.377475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:53.377579 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:53.377625 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:53.377950 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 08:58:53.378013 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 08:58:53.378374 16113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:53.369186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.2:57123)
0619 08:58:53.369348 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:53.369384 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:53.369875 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:53.370136 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:53.370139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:53.370707 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:58:53.370899 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:53.370904 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.371576 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.371577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:53.372516 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:53.372520 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.373001 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.373002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:53.373155 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:53.373766 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:58:53.373776 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:58:53.375260 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:58:53.376721 (+ 1461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:53.376729 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:53.376732 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:53.377084 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:53.377271 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:53.377272 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:53.377273 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:53.377416 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:53.377634 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:58:53.377640 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:58:53.377820 (+ 180us) client_negotiation.cc:769] Sending connection context
0619 08:58:53.377950 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 08:58:53.378132 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":84,"threads_started":1}
I20260619 08:58:53.378865 16111 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41155
I20260619 08:58:53.378979 16111 heartbeater.cc:464] Registering TS with master...
I20260619 08:58:53.379182 16111 heartbeater.cc:511] Master 127.0.98.62:41155 requested a full tablet report, sending...
I20260619 08:58:53.379573 15786 ts_manager.cc:194] Registered new tserver with Master: decab1e0f88a49b9922817b18178c8d2 (127.0.98.2:45001)
I20260619 08:58:53.379944 15786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57123
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 08:58:53.396891 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41155
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:58:53.475991 16116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:58:53.476263 16116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:58:53.476300 16116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:58:53.477490 16116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:58:53.477535 16116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:58:53.477581 16116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 08:58:53.479036 16116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:58:53.479282 16116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:58:53.479496 16116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:58:53.484226 16124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:58:53.484242 16121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:53.484319 16116 server_base.cc:1034] running on GCE node
W20260619 08:58:53.484287 16122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:58:53.484687 16116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:58:53.484890 16116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:58:53.486054 16116 hybrid_clock.cc:648] HybridClock initialized: now 1781859533486040 us; error 26 us; skew 500 ppm
Jun 19 08:58:53 dist-test-slave-h4g5 krb5kdc[15741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:58:53.488657 16116 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 08:58:53.489091 16116 webserver.cc:460] Webserver started at http://127.0.98.3:37297/ using document root <none> and password file <none>
I20260619 08:58:53.489300 16116 fs_manager.cc:359] Metadata directory not provided
I20260619 08:58:53.489348 16116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:58:53.489487 16116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:58:53.490552 16116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/data/instance:
uuid: "83a3420a7ea94a53a4a2011b576f7a2d"
format_stamp: "Formatted at 2026-06-19 08:58:53 on dist-test-slave-h4g5"
server_key: "d4da8bc4f318a166b70b7508a05b0d70"
server_key_iv: "75369f20d790b1dcdda4d29d7e5537c4"
server_key_version: "encryptionkey@0"
I20260619 08:58:53.490877 16116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/wal/instance:
uuid: "83a3420a7ea94a53a4a2011b576f7a2d"
format_stamp: "Formatted at 2026-06-19 08:58:53 on dist-test-slave-h4g5"
server_key: "d4da8bc4f318a166b70b7508a05b0d70"
server_key_iv: "75369f20d790b1dcdda4d29d7e5537c4"
server_key_version: "encryptionkey@0"
I20260619 08:58:53.492113 16116 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 08:58:53.492797 16131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:53.492941 16116 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 08:58:53.493026 16116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/wal
uuid: "83a3420a7ea94a53a4a2011b576f7a2d"
format_stamp: "Formatted at 2026-06-19 08:58:53 on dist-test-slave-h4g5"
server_key: "d4da8bc4f318a166b70b7508a05b0d70"
server_key_iv: "75369f20d790b1dcdda4d29d7e5537c4"
server_key_version: "encryptionkey@0"
I20260619 08:58:53.493093 16116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:58:53.508219 16116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:58:53.510537 16116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:58:53.510672 16116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:58:53.510908 16116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:58:53.511212 16116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:58:53.511245 16116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:53.511273 16116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:58:53.511291 16116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:58:53.516880 16116 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32805
I20260619 08:58:53.516944 16244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32805 every 8 connection(s)
I20260619 08:58:53.517233 16116 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
I20260619 08:58:53.521476 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 16116
I20260619 08:58:53.521572 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/wal/instance
I20260619 08:58:53.521746 392 external_mini_cluster.cc:1428] Setting key fef0a1eed9328b4c9d215f228a71275a
Jun 19 08:58:53 dist-test-slave-h4g5 krb5kdc[15741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:58:53.525722 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:53.517923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38023 (local address 127.0.98.62:41155)
0619 08:58:53.517976 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:53.517978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:53.518166 (+ 188us) server_negotiation.cc:407] Connection header received
0619 08:58:53.518629 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:53.518629 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:53.518638 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:53.518694 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:53.519586 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.520166 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.521138 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.521356 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.521430 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:53.523420 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:53.523421 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:53.523422 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:53.523428 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:53.524601 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:53.524988 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:53.524989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:53.524989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:53.525036 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:53.525222 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:53.525223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:53.525223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:53.525318 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:53.525362 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:53.525600 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 08:58:53.525658 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 08:58:53.525885 16247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:53.517895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.3:38023)
0619 08:58:53.518043 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:53.518085 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:53.518553 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:53.518752 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:53.518755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:53.519311 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:58:53.519508 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:53.519513 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.520235 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.520235 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:53.521050 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:53.521052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.521426 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.521426 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:53.521525 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:53.522051 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:58:53.522060 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:58:53.523339 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:58:53.524691 (+ 1352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:53.524697 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:53.524699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:53.524923 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:53.525080 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:58:53.525081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:58:53.525081 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:58:53.525179 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:58:53.525360 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:58:53.525363 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:58:53.525481 (+ 118us) client_negotiation.cc:769] Sending connection context
0619 08:58:53.525567 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 08:58:53.525688 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
I20260619 08:58:53.526310 16245 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41155
I20260619 08:58:53.526398 16245 heartbeater.cc:464] Registering TS with master...
I20260619 08:58:53.526544 16245 heartbeater.cc:511] Master 127.0.98.62:41155 requested a full tablet report, sending...
I20260619 08:58:53.526863 15786 ts_manager.cc:194] Registered new tserver with Master: 83a3420a7ea94a53a4a2011b576f7a2d (127.0.98.3:32805)
I20260619 08:58:53.527199 15786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38023
I20260619 08:58:53.534154 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:58:53.541455 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:53.535424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33674 (local address 127.0.98.62:41155)
0619 08:58:53.535476 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:53.535478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:53.535575 (+ 97us) server_negotiation.cc:407] Connection header received
0619 08:58:53.535623 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:53.535623 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:53.535632 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:53.535683 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:58:53.536259 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.536768 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.537562 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.537751 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.537805 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:53.539092 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:58:53.539094 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:58:53.539094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:58:53.539100 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:58:53.540411 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:53.540760 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:53.540761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:53.540761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:53.540808 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:58:53.540954 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:58:53.540954 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:58:53.540955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:58:53.541102 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:58:53.541173 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:53.541324 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 08:58:53.541389 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 08:58:53.543121 15786 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:58:53.544811 15786 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:58:53.551802 16253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:53.546895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32805 (local address 127.0.0.1:34500)
0619 08:58:53.547087 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:53.547132 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:53.547196 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:53.547704 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:53.547707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:53.547709 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:58:53.547781 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:53.547782 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.549144 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.549145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:53.550510 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:53.550513 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.551483 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.551484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:53.551580 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:53.551596 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 08:58:53.551666 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 08:58:53.551719 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":66,"threads_started":1}
I20260619 08:58:53.551810 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:53.547347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34500 (local address 127.0.98.3:32805)
0619 08:58:53.547544 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:53.547547 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:53.547555 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:58:53.547586 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:53.547588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:53.547600 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:53.547669 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:58:53.547842 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.549055 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.550636 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.551403 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.551482 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:53.551542 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:53.551682 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 08:58:53.551719 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":119,"threads_started":1}
I20260619 08:58:53.552686 16179 tablet_service.cc:1467] Processing CreateTablet for tablet 322261cd38b64fb2abd5225ed992ef35 (DEFAULT_TABLE table=test-table [id=4b8fc71abc2d4352b488654978b03d38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:58:53.552958 16255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:53.547103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40207 (local address 127.0.0.1:50156)
0619 08:58:53.549187 (+ 2084us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:53.549198 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:53.549245 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:53.549458 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:53.549459 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:53.549461 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:58:53.549508 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:53.549510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.550730 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.550731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:53.551848 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:53.551850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.552644 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.552645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:53.552706 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:53.552710 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:58:53.552777 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 08:58:53.552813 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2071,"thread_start_us":36,"threads_started":1}
I20260619 08:58:53.552974 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:53.547229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50156 (local address 127.0.98.1:40207)
0619 08:58:53.547382 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:53.547386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:53.549265 (+ 1879us) server_negotiation.cc:407] Connection header received
0619 08:58:53.549334 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:53.549337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:53.549353 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:53.549459 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:58:53.549555 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.550632 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.551939 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.552566 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.552622 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:53.552657 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:53.552851 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 08:58:53.552892 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20260619 08:58:53.553100 16179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 322261cd38b64fb2abd5225ed992ef35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:58:53.553579 15911 tablet_service.cc:1467] Processing CreateTablet for tablet 322261cd38b64fb2abd5225ed992ef35 (DEFAULT_TABLE table=test-table [id=4b8fc71abc2d4352b488654978b03d38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:58:53.553851 15911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 322261cd38b64fb2abd5225ed992ef35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:58:53.555968 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:53.546896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45001 (local address 127.0.0.1:51182)
0619 08:58:53.547181 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:53.547190 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:53.547226 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:53.547795 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:53.547797 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:53.547799 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:58:53.547853 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:53.547855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.550639 (+ 2784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.550640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:53.551356 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:53.551358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.555731 (+ 4373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.555732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:53.555800 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:53.555804 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:58:53.555856 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 08:58:53.555898 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":2,"spinlock_wait_cycles":768,"thread_start_us":63,"threads_started":1}
I20260619 08:58:53.556005 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:53.547157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51182 (local address 127.0.98.2:45001)
0619 08:58:53.547577 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:53.547581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:53.547590 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:58:53.547622 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:53.547623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:53.547635 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:53.547713 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:58:53.547931 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.549061 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.551439 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.552202 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.552257 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:53.555692 (+ 3435us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:53.555883 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 08:58:53.555931 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":88,"threads_started":1}
I20260619 08:58:53.556442 16260 tablet_bootstrap.cc:492] T 322261cd38b64fb2abd5225ed992ef35 P 28eabc22c4e04f059339795b02f88ed5: Bootstrap starting.
I20260619 08:58:53.556648 16045 tablet_service.cc:1467] Processing CreateTablet for tablet 322261cd38b64fb2abd5225ed992ef35 (DEFAULT_TABLE table=test-table [id=4b8fc71abc2d4352b488654978b03d38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:58:53.556916 16045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 322261cd38b64fb2abd5225ed992ef35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:58:53.556995 16260 tablet_bootstrap.cc:654] T 322261cd38b64fb2abd5225ed992ef35 P 28eabc22c4e04f059339795b02f88ed5: Neither blocks nor log segments found. Creating new log.
I20260619 08:58:53.557439 16260 log.cc:824] T 322261cd38b64fb2abd5225ed992ef35 P 28eabc22c4e04f059339795b02f88ed5: Log is configured to *not* fsync() on all Append() calls
I20260619 08:58:53.558059 16260 tablet_bootstrap.cc:492] T 322261cd38b64fb2abd5225ed992ef35 P 28eabc22c4e04f059339795b02f88ed5: No bootstrap required, opened a new log
I20260619 08:58:53.558118 16260 ts_tablet_manager.cc:1397] T 322261cd38b64fb2abd5225ed992ef35 P 28eabc22c4e04f059339795b02f88ed5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:58:53.559229 16262 tablet_bootstrap.cc:492] T 322261cd38b64fb2abd5225ed992ef35 P decab1e0f88a49b9922817b18178c8d2: Bootstrap starting.
I20260619 08:58:53.559248 16259 tablet_bootstrap.cc:492] T 322261cd38b64fb2abd5225ed992ef35 P 83a3420a7ea94a53a4a2011b576f7a2d: Bootstrap starting.
I20260619 08:58:53.559545 16260 raft_consensus.cc:348] T 322261cd38b64fb2abd5225ed992ef35 P 28eabc22c4e04f059339795b02f88ed5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32805 } } peers { permanent_uuid: "decab1e0f88a49b9922817b18178c8d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "28eabc22c4e04f059339795b02f88ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40207 } }
I20260619 08:58:53.559720 16260 raft_consensus.cc:374] T 322261cd38b64fb2abd5225ed992ef35 P 28eabc22c4e04f059339795b02f88ed5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:58:53.559759 16260 raft_consensus.cc:729] T 322261cd38b64fb2abd5225ed992ef35 P 28eabc22c4e04f059339795b02f88ed5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28eabc22c4e04f059339795b02f88ed5, State: Initialized, Role: FOLLOWER
I20260619 08:58:53.559890 16262 tablet_bootstrap.cc:654] T 322261cd38b64fb2abd5225ed992ef35 P decab1e0f88a49b9922817b18178c8d2: Neither blocks nor log segments found. Creating new log.
I20260619 08:58:53.559890 16259 tablet_bootstrap.cc:654] T 322261cd38b64fb2abd5225ed992ef35 P 83a3420a7ea94a53a4a2011b576f7a2d: Neither blocks nor log segments found. Creating new log.
I20260619 08:58:53.559893 16260 consensus_queue.cc:260] T 322261cd38b64fb2abd5225ed992ef35 P 28eabc22c4e04f059339795b02f88ed5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32805 } } peers { permanent_uuid: "decab1e0f88a49b9922817b18178c8d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "28eabc22c4e04f059339795b02f88ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40207 } }
I20260619 08:58:53.560195 16260 ts_tablet_manager.cc:1428] T 322261cd38b64fb2abd5225ed992ef35 P 28eabc22c4e04f059339795b02f88ed5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:58:53.560230 15977 heartbeater.cc:503] Master 127.0.98.62:41155 was elected leader, sending a full tablet report...
I20260619 08:58:53.560484 16259 log.cc:824] T 322261cd38b64fb2abd5225ed992ef35 P 83a3420a7ea94a53a4a2011b576f7a2d: Log is configured to *not* fsync() on all Append() calls
I20260619 08:58:53.560482 16262 log.cc:824] T 322261cd38b64fb2abd5225ed992ef35 P decab1e0f88a49b9922817b18178c8d2: Log is configured to *not* fsync() on all Append() calls
I20260619 08:58:53.561270 16259 tablet_bootstrap.cc:492] T 322261cd38b64fb2abd5225ed992ef35 P 83a3420a7ea94a53a4a2011b576f7a2d: No bootstrap required, opened a new log
I20260619 08:58:53.561338 16259 ts_tablet_manager.cc:1397] T 322261cd38b64fb2abd5225ed992ef35 P 83a3420a7ea94a53a4a2011b576f7a2d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 08:58:53.561429 16262 tablet_bootstrap.cc:492] T 322261cd38b64fb2abd5225ed992ef35 P decab1e0f88a49b9922817b18178c8d2: No bootstrap required, opened a new log
I20260619 08:58:53.561493 16262 ts_tablet_manager.cc:1397] T 322261cd38b64fb2abd5225ed992ef35 P decab1e0f88a49b9922817b18178c8d2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 08:58:53.562700 16259 raft_consensus.cc:348] T 322261cd38b64fb2abd5225ed992ef35 P 83a3420a7ea94a53a4a2011b576f7a2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32805 } } peers { permanent_uuid: "decab1e0f88a49b9922817b18178c8d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "28eabc22c4e04f059339795b02f88ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40207 } }
I20260619 08:58:53.562835 16259 raft_consensus.cc:374] T 322261cd38b64fb2abd5225ed992ef35 P 83a3420a7ea94a53a4a2011b576f7a2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:58:53.562857 16259 raft_consensus.cc:729] T 322261cd38b64fb2abd5225ed992ef35 P 83a3420a7ea94a53a4a2011b576f7a2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83a3420a7ea94a53a4a2011b576f7a2d, State: Initialized, Role: FOLLOWER
I20260619 08:58:53.562783 16262 raft_consensus.cc:348] T 322261cd38b64fb2abd5225ed992ef35 P decab1e0f88a49b9922817b18178c8d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32805 } } peers { permanent_uuid: "decab1e0f88a49b9922817b18178c8d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "28eabc22c4e04f059339795b02f88ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40207 } }
I20260619 08:58:53.562927 16262 raft_consensus.cc:374] T 322261cd38b64fb2abd5225ed992ef35 P decab1e0f88a49b9922817b18178c8d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:58:53.562965 16262 raft_consensus.cc:729] T 322261cd38b64fb2abd5225ed992ef35 P decab1e0f88a49b9922817b18178c8d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: decab1e0f88a49b9922817b18178c8d2, State: Initialized, Role: FOLLOWER
I20260619 08:58:53.562955 16259 consensus_queue.cc:260] T 322261cd38b64fb2abd5225ed992ef35 P 83a3420a7ea94a53a4a2011b576f7a2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32805 } } peers { permanent_uuid: "decab1e0f88a49b9922817b18178c8d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "28eabc22c4e04f059339795b02f88ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40207 } }
I20260619 08:58:53.563072 16262 consensus_queue.cc:260] T 322261cd38b64fb2abd5225ed992ef35 P decab1e0f88a49b9922817b18178c8d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32805 } } peers { permanent_uuid: "decab1e0f88a49b9922817b18178c8d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "28eabc22c4e04f059339795b02f88ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40207 } }
I20260619 08:58:53.563222 16245 heartbeater.cc:503] Master 127.0.98.62:41155 was elected leader, sending a full tablet report...
I20260619 08:58:53.563326 16262 ts_tablet_manager.cc:1428] T 322261cd38b64fb2abd5225ed992ef35 P decab1e0f88a49b9922817b18178c8d2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 08:58:53.563352 16111 heartbeater.cc:503] Master 127.0.98.62:41155 was elected leader, sending a full tablet report...
I20260619 08:58:53.563453 16259 ts_tablet_manager.cc:1428] T 322261cd38b64fb2abd5225ed992ef35 P 83a3420a7ea94a53a4a2011b576f7a2d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260619 08:58:53.619097 16112 tablet.cc:2369] T 322261cd38b64fb2abd5225ed992ef35 P decab1e0f88a49b9922817b18178c8d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:58:53.640777 16266 raft_consensus.cc:482] T 322261cd38b64fb2abd5225ed992ef35 P 83a3420a7ea94a53a4a2011b576f7a2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:58:53.640962 16266 raft_consensus.cc:504] T 322261cd38b64fb2abd5225ed992ef35 P 83a3420a7ea94a53a4a2011b576f7a2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32805 } } peers { permanent_uuid: "decab1e0f88a49b9922817b18178c8d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "28eabc22c4e04f059339795b02f88ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40207 } }
I20260619 08:58:53.641332 16266 leader_election.cc:288] T 322261cd38b64fb2abd5225ed992ef35 P 83a3420a7ea94a53a4a2011b576f7a2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers decab1e0f88a49b9922817b18178c8d2 (127.0.98.2:45001), 28eabc22c4e04f059339795b02f88ed5 (127.0.98.1:40207)
I20260619 08:58:53.645115 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:53.641595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34649 (local address 127.0.98.1:40207)
0619 08:58:53.641646 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:53.641648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:53.641657 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:58:53.641686 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:53.641687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:53.641697 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:53.641755 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:58:53.641912 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.642672 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.644176 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.644735 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.644808 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:53.644817 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:53.644990 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 08:58:53.645028 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 08:58:53.645115 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:53.641620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35697 (local address 127.0.98.2:45001)
0619 08:58:53.641660 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:53.641663 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:53.641789 (+ 126us) server_negotiation.cc:407] Connection header received
0619 08:58:53.641820 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:53.641821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:53.641831 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:53.641871 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:58:53.642022 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.642739 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.644163 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.644735 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.644808 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:53.644817 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:53.645007 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 08:58:53.645045 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 08:58:53.645120 16247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:53.641512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40207 (local address 127.0.98.3:34649)
0619 08:58:53.641566 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:53.641578 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:53.641619 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:53.641775 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:53.641776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:53.641778 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:58:53.641847 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:53.641849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.642777 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.642778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:53.644064 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:53.644067 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.644816 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.644817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:53.644921 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:53.644923 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:58:53.644979 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 08:58:53.645026 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":67}
I20260619 08:58:53.645129 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:53.641512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45001 (local address 127.0.98.3:35697)
0619 08:58:53.641725 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:53.641732 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 08:58:53.641782 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:58:53.641907 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:58:53.641908 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:58:53.641909 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:58:53.641969 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:53.641970 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.642814 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.642814 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:53.644065 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:58:53.644067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.644816 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.644817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:58:53.644916 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:53.644921 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 08:58:53.644987 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 08:58:53.645026 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":24,"thread_start_us":126,"threads_started":1}
I20260619 08:58:53.645488 15931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "322261cd38b64fb2abd5225ed992ef35" candidate_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28eabc22c4e04f059339795b02f88ed5" is_pre_election: true
I20260619 08:58:53.645654 15931 raft_consensus.cc:2454] T 322261cd38b64fb2abd5225ed992ef35 P 28eabc22c4e04f059339795b02f88ed5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83a3420a7ea94a53a4a2011b576f7a2d in term 0.
I20260619 08:58:53.645618 16065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "322261cd38b64fb2abd5225ed992ef35" candidate_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decab1e0f88a49b9922817b18178c8d2" is_pre_election: true
I20260619 08:58:53.645740 16065 raft_consensus.cc:2454] T 322261cd38b64fb2abd5225ed992ef35 P decab1e0f88a49b9922817b18178c8d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83a3420a7ea94a53a4a2011b576f7a2d in term 0.
I20260619 08:58:53.645889 16132 leader_election.cc:302] T 322261cd38b64fb2abd5225ed992ef35 P 83a3420a7ea94a53a4a2011b576f7a2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28eabc22c4e04f059339795b02f88ed5, 83a3420a7ea94a53a4a2011b576f7a2d; no voters:
I20260619 08:58:53.646039 16266 raft_consensus.cc:2781] T 322261cd38b64fb2abd5225ed992ef35 P 83a3420a7ea94a53a4a2011b576f7a2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:58:53.646109 16266 raft_consensus.cc:482] T 322261cd38b64fb2abd5225ed992ef35 P 83a3420a7ea94a53a4a2011b576f7a2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:58:53.646135 16266 raft_consensus.cc:3037] T 322261cd38b64fb2abd5225ed992ef35 P 83a3420a7ea94a53a4a2011b576f7a2d [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:58:53.646808 16266 raft_consensus.cc:504] T 322261cd38b64fb2abd5225ed992ef35 P 83a3420a7ea94a53a4a2011b576f7a2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32805 } } peers { permanent_uuid: "decab1e0f88a49b9922817b18178c8d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "28eabc22c4e04f059339795b02f88ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40207 } }
I20260619 08:58:53.646927 16266 leader_election.cc:288] T 322261cd38b64fb2abd5225ed992ef35 P 83a3420a7ea94a53a4a2011b576f7a2d [CANDIDATE]: Term 1 election: Requested vote from peers decab1e0f88a49b9922817b18178c8d2 (127.0.98.2:45001), 28eabc22c4e04f059339795b02f88ed5 (127.0.98.1:40207)
I20260619 08:58:53.647078 16065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "322261cd38b64fb2abd5225ed992ef35" candidate_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decab1e0f88a49b9922817b18178c8d2"
I20260619 08:58:53.647150 16065 raft_consensus.cc:3037] T 322261cd38b64fb2abd5225ed992ef35 P decab1e0f88a49b9922817b18178c8d2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:58:53.647821 16065 raft_consensus.cc:2454] T 322261cd38b64fb2abd5225ed992ef35 P decab1e0f88a49b9922817b18178c8d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83a3420a7ea94a53a4a2011b576f7a2d in term 1.
I20260619 08:58:53.647919 15931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "322261cd38b64fb2abd5225ed992ef35" candidate_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28eabc22c4e04f059339795b02f88ed5"
I20260619 08:58:53.647989 16133 leader_election.cc:302] T 322261cd38b64fb2abd5225ed992ef35 P 83a3420a7ea94a53a4a2011b576f7a2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83a3420a7ea94a53a4a2011b576f7a2d, decab1e0f88a49b9922817b18178c8d2; no voters:
I20260619 08:58:53.648010 15931 raft_consensus.cc:3037] T 322261cd38b64fb2abd5225ed992ef35 P 28eabc22c4e04f059339795b02f88ed5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:58:53.648126 16266 raft_consensus.cc:2781] T 322261cd38b64fb2abd5225ed992ef35 P 83a3420a7ea94a53a4a2011b576f7a2d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:58:53.648283 16266 raft_consensus.cc:686] T 322261cd38b64fb2abd5225ed992ef35 P 83a3420a7ea94a53a4a2011b576f7a2d [term 1 LEADER]: Becoming Leader. State: Replica: 83a3420a7ea94a53a4a2011b576f7a2d, State: Running, Role: LEADER
I20260619 08:58:53.648397 16266 consensus_queue.cc:237] T 322261cd38b64fb2abd5225ed992ef35 P 83a3420a7ea94a53a4a2011b576f7a2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32805 } } peers { permanent_uuid: "decab1e0f88a49b9922817b18178c8d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "28eabc22c4e04f059339795b02f88ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40207 } }
I20260619 08:58:53.648831 15931 raft_consensus.cc:2454] T 322261cd38b64fb2abd5225ed992ef35 P 28eabc22c4e04f059339795b02f88ed5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83a3420a7ea94a53a4a2011b576f7a2d in term 1.
I20260619 08:58:53.649199 15786 catalog_manager.cc:5495] T 322261cd38b64fb2abd5225ed992ef35 P 83a3420a7ea94a53a4a2011b576f7a2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 83a3420a7ea94a53a4a2011b576f7a2d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "decab1e0f88a49b9922817b18178c8d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28eabc22c4e04f059339795b02f88ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40207 } health_report { overall_health: UNKNOWN } } }
I20260619 08:58:53.667368 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:53.665104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34514 (local address 127.0.98.3:32805)
0619 08:58:53.665159 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 08:58:53.665161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:58:53.665171 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:58:53.665236 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:58:53.665237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:58:53.665260 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:58:53.665321 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:58:53.665444 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.665913 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.666673 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:58:53.666872 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:58:53.666925 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:58:53.667079 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:58:53.667180 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:58:53.667250 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 08:58:53.667273 (+ 23us) server_negotiation.cc:299] Negotiation successful
0619 08:58:53.667293 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 08:58:53.669134 15931 raft_consensus.cc:1261] T 322261cd38b64fb2abd5225ed992ef35 P 28eabc22c4e04f059339795b02f88ed5 [term 1 FOLLOWER]: Refusing update from remote peer 83a3420a7ea94a53a4a2011b576f7a2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:58:53.669188 16065 raft_consensus.cc:1261] T 322261cd38b64fb2abd5225ed992ef35 P decab1e0f88a49b9922817b18178c8d2 [term 1 FOLLOWER]: Refusing update from remote peer 83a3420a7ea94a53a4a2011b576f7a2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:58:53.669492 16266 consensus_queue.cc:1035] T 322261cd38b64fb2abd5225ed992ef35 P 83a3420a7ea94a53a4a2011b576f7a2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "28eabc22c4e04f059339795b02f88ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:58:53.669628 16269 consensus_queue.cc:1035] T 322261cd38b64fb2abd5225ed992ef35 P 83a3420a7ea94a53a4a2011b576f7a2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "decab1e0f88a49b9922817b18178c8d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:58:53.671981 16272 mvcc.cc:205] Tried to move back new op lower bound from 7298496649906077696 to 7298496649824448512. Current Snapshot: MvccSnapshot[applied={T|T < 7298496649906077696}]
I20260619 08:58:53.672106 16274 mvcc.cc:205] Tried to move back new op lower bound from 7298496649906077696 to 7298496649824448512. Current Snapshot: MvccSnapshot[applied={T|T < 7298496649906077696}]
I20260619 08:58:53.672384 16273 mvcc.cc:205] Tried to move back new op lower bound from 7298496649906077696 to 7298496649824448512. Current Snapshot: MvccSnapshot[applied={T|T < 7298496649906077696}]
I20260619 08:58:53.674664 15786 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:58:53.674758 15786 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:58:53.675458 15786 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b: Sending DeleteTablet for 3 replicas of tablet 322261cd38b64fb2abd5225ed992ef35
I20260619 08:58:53.675791 16045 tablet_service.cc:1514] Processing DeleteTablet for tablet 322261cd38b64fb2abd5225ed992ef35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:58:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51182
I20260619 08:58:53.675792 16179 tablet_service.cc:1514] Processing DeleteTablet for tablet 322261cd38b64fb2abd5225ed992ef35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:58:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34500
I20260619 08:58:53.675817 15911 tablet_service.cc:1514] Processing DeleteTablet for tablet 322261cd38b64fb2abd5225ed992ef35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:58:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50156
I20260619 08:58:53.676057 16284 tablet_replica.cc:330] T 322261cd38b64fb2abd5225ed992ef35 P 28eabc22c4e04f059339795b02f88ed5: stopping tablet replica
I20260619 08:58:53.676126 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 15848
I20260619 08:58:53.676190 16284 raft_consensus.cc:2229] T 322261cd38b64fb2abd5225ed992ef35 P 28eabc22c4e04f059339795b02f88ed5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:58:53.676286 16283 tablet_replica.cc:330] T 322261cd38b64fb2abd5225ed992ef35 P 83a3420a7ea94a53a4a2011b576f7a2d: stopping tablet replica
I20260619 08:58:53.676398 16283 raft_consensus.cc:2229] T 322261cd38b64fb2abd5225ed992ef35 P 83a3420a7ea94a53a4a2011b576f7a2d [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:58:53.676535 16283 raft_consensus.cc:2258] T 322261cd38b64fb2abd5225ed992ef35 P 83a3420a7ea94a53a4a2011b576f7a2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:58:53.676985 16283 ts_tablet_manager.cc:1905] T 322261cd38b64fb2abd5225ed992ef35 P 83a3420a7ea94a53a4a2011b576f7a2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:58:53.678386 16282 tablet_replica.cc:330] T 322261cd38b64fb2abd5225ed992ef35 P decab1e0f88a49b9922817b18178c8d2: stopping tablet replica
I20260619 08:58:53.678503 16282 raft_consensus.cc:2229] T 322261cd38b64fb2abd5225ed992ef35 P decab1e0f88a49b9922817b18178c8d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:58:53.678610 16282 raft_consensus.cc:2258] T 322261cd38b64fb2abd5225ed992ef35 P decab1e0f88a49b9922817b18178c8d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:58:53.678747 16283 ts_tablet_manager.cc:1918] T 322261cd38b64fb2abd5225ed992ef35 P 83a3420a7ea94a53a4a2011b576f7a2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:58:53.678980 16283 log.cc:1196] T 322261cd38b64fb2abd5225ed992ef35 P 83a3420a7ea94a53a4a2011b576f7a2d: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/wal/wals/322261cd38b64fb2abd5225ed992ef35
I20260619 08:58:53.679030 16282 ts_tablet_manager.cc:1905] T 322261cd38b64fb2abd5225ed992ef35 P decab1e0f88a49b9922817b18178c8d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:58:53.679342 16283 ts_tablet_manager.cc:1939] T 322261cd38b64fb2abd5225ed992ef35 P 83a3420a7ea94a53a4a2011b576f7a2d: Deleting consensus metadata
I20260619 08:58:53.679788 15770 catalog_manager.cc:4841] TS 83a3420a7ea94a53a4a2011b576f7a2d (127.0.98.3:32805): tablet 322261cd38b64fb2abd5225ed992ef35 (table test-table [id=4b8fc71abc2d4352b488654978b03d38]) successfully deleted
I20260619 08:58:53.680420 16282 ts_tablet_manager.cc:1918] T 322261cd38b64fb2abd5225ed992ef35 P decab1e0f88a49b9922817b18178c8d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:58:53.680481 16282 log.cc:1196] T 322261cd38b64fb2abd5225ed992ef35 P decab1e0f88a49b9922817b18178c8d2: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/wal/wals/322261cd38b64fb2abd5225ed992ef35
I20260619 08:58:53.680696 16282 ts_tablet_manager.cc:1939] T 322261cd38b64fb2abd5225ed992ef35 P decab1e0f88a49b9922817b18178c8d2: Deleting consensus metadata
I20260619 08:58:53.681100 15771 catalog_manager.cc:4841] TS decab1e0f88a49b9922817b18178c8d2 (127.0.98.2:45001): tablet 322261cd38b64fb2abd5225ed992ef35 (table test-table [id=4b8fc71abc2d4352b488654978b03d38]) successfully deleted
W20260619 08:58:53.682143 15770 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40207 (error 108)
I20260619 08:58:53.682266 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 15982
I20260619 08:58:53.683202 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:58:53.683061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40207 (local address 127.0.0.1:50162)
0619 08:58:53.683121 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 08:58:53.683167 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 08:58:53.683336 15770 catalog_manager.cc:4584] TS 28eabc22c4e04f059339795b02f88ed5 (127.0.98.1:40207): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 322261cd38b64fb2abd5225ed992ef35: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40207: connect: Connection refused (error 111)
I20260619 08:58:53.687079 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 16116
I20260619 08:58:53.691401 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 15754
I20260619 08:59:13.695973 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41155
--webserver_interface=127.0.98.62
--webserver_port=40485
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:59:13.777096 16286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:59:13.777309 16286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:59:13.777339 16286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:59:13.778504 16286 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 08:59:13.778542 16286 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 08:59:13.778559 16286 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 08:59:13.778574 16286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:59:13.778590 16286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:59:13.778609 16286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:59:13.778628 16286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:59:13.778652 16286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:59:13.780225 16286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:59:13.780469 16286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:59:13.780681 16286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:59:13.789227 16295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:13.789211 16293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:13.789247 16286 server_base.cc:1034] running on GCE node
W20260619 08:59:13.789206 16292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:13.789755 16286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:59:13.790077 16286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:59:13.791230 16286 hybrid_clock.cc:648] HybridClock initialized: now 1781859553791198 us; error 46 us; skew 500 ppm
Jun 19 08:59:13 dist-test-slave-h4g5 krb5kdc[15741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:59:13.793721 16286 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 08:59:13.794194 16286 webserver.cc:460] Webserver started at http://127.0.98.62:40485/ using document root <none> and password file <none>
I20260619 08:59:13.794415 16286 fs_manager.cc:359] Metadata directory not provided
I20260619 08:59:13.794477 16286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:59:13.795868 16286 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:59:13.796548 16302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:13.796693 16286 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:13.796780 16286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "19abdfe959a1427f91d2c92fa822266b"
format_stamp: "Formatted at 2026-06-19 08:58:53 on dist-test-slave-h4g5"
server_key: "39e9d43f289b6845e9ce204af44cd31c"
server_key_iv: "01c7fda7edd719f6f86a5c354ff89914"
server_key_version: "encryptionkey@0"
I20260619 08:59:13.797171 16286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:59:13.810099 16286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:59:13.812067 16286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:59:13.812250 16286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:59:13.816043 16286 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41155
I20260619 08:59:13.816155 16354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41155 every 8 connection(s)
I20260619 08:59:13.816511 16286 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:59:13.818040 16355 sys_catalog.cc:263] Verifying existing consensus state
I20260619 08:59:13.818712 16355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b: Bootstrap starting.
I20260619 08:59:13.820760 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 16286
I20260619 08:59:13.821033 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40207
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41155
--webserver_port=39765
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 08:59:13.827399 16355 log.cc:824] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b: Log is configured to *not* fsync() on all Append() calls
I20260619 08:59:13.829583 16355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 08:59:13.829876 16355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b: Bootstrap complete.
I20260619 08:59:13.832108 16355 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19abdfe959a1427f91d2c92fa822266b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41155 } }
I20260619 08:59:13.832405 16355 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19abdfe959a1427f91d2c92fa822266b, State: Initialized, Role: FOLLOWER
I20260619 08:59:13.832576 16355 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19abdfe959a1427f91d2c92fa822266b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41155 } }
I20260619 08:59:13.832657 16355 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:59:13.832717 16355 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:59:13.832769 16355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [term 1 FOLLOWER]: Advancing to term 2
I20260619 08:59:13.833858 16355 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19abdfe959a1427f91d2c92fa822266b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41155 } }
I20260619 08:59:13.833985 16355 leader_election.cc:302] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19abdfe959a1427f91d2c92fa822266b; no voters:
I20260619 08:59:13.834174 16355 leader_election.cc:288] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 08:59:13.834214 16359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [term 2 FOLLOWER]: Leader election won for term 2
I20260619 08:59:13.834416 16359 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [term 2 LEADER]: Becoming Leader. State: Replica: 19abdfe959a1427f91d2c92fa822266b, State: Running, Role: LEADER
I20260619 08:59:13.834519 16355 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:59:13.834509 16359 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19abdfe959a1427f91d2c92fa822266b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41155 } }
I20260619 08:59:13.834810 16360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "19abdfe959a1427f91d2c92fa822266b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19abdfe959a1427f91d2c92fa822266b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41155 } } }
I20260619 08:59:13.834921 16360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [sys.catalog]: This master's current role is: LEADER
I20260619 08:59:13.835156 16361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19abdfe959a1427f91d2c92fa822266b. Latest consensus state: current_term: 2 leader_uuid: "19abdfe959a1427f91d2c92fa822266b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19abdfe959a1427f91d2c92fa822266b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41155 } } }
I20260619 08:59:13.835251 16361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [sys.catalog]: This master's current role is: LEADER
I20260619 08:59:13.835282 16365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:59:13.836220 16365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:59:13.836335 16365 catalog_manager.cc:1259] Loaded cluster ID: 702f520165994b689af549353709df7b
I20260619 08:59:13.836373 16365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:59:13.837502 16365 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:59:13.837704 16365 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b: Loaded TSK: 0
I20260619 08:59:13.854986 16365 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b: Generated new TSK 1
I20260619 08:59:13.855108 16365 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 08:59:13.902664 16357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:59:13.902870 16357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:59:13.902894 16357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:59:13.904053 16357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:59:13.904120 16357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:59:13.904162 16357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 08:59:13.905619 16357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40207
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39765
--tserver_master_addrs=127.0.98.62:41155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:59:13.905833 16357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:59:13.906051 16357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:59:13.910974 16382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:13.911026 16383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:13.910987 16385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:13.911465 16357 server_base.cc:1034] running on GCE node
I20260619 08:59:13.911689 16357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:59:13.911999 16357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:59:13.913188 16357 hybrid_clock.cc:648] HybridClock initialized: now 1781859553913163 us; error 35 us; skew 500 ppm
Jun 19 08:59:13 dist-test-slave-h4g5 krb5kdc[15741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:59:13.916288 16357 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 08:59:13.916887 16357 webserver.cc:460] Webserver started at http://127.0.98.1:39765/ using document root <none> and password file <none>
I20260619 08:59:13.917177 16357 fs_manager.cc:359] Metadata directory not provided
I20260619 08:59:13.917273 16357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:59:13.919167 16357 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:59:13.919902 16392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:13.920099 16357 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:59:13.920176 16357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/wal
uuid: "28eabc22c4e04f059339795b02f88ed5"
format_stamp: "Formatted at 2026-06-19 08:58:53 on dist-test-slave-h4g5"
server_key: "263490f766468d0e26e514905f24c728"
server_key_iv: "a746b61d4dac9f74b82e5c0460297042"
server_key_version: "encryptionkey@0"
I20260619 08:59:13.920519 16357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:59:13.933203 16357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:59:13.935536 16357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:59:13.935685 16357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:59:13.935950 16357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:59:13.936425 16399 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 08:59:13.938290 16357 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 08:59:13.938337 16357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 08:59:13.938372 16357 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 08:59:13.938975 16357 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 08:59:13.939010 16357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 08:59:13.939096 16399 tablet_bootstrap.cc:492] T 322261cd38b64fb2abd5225ed992ef35 P 28eabc22c4e04f059339795b02f88ed5: Bootstrap starting.
I20260619 08:59:13.945474 16357 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40207
I20260619 08:59:13.945850 16357 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
I20260619 08:59:13.948372 16506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40207 every 8 connection(s)
I20260619 08:59:13.949956 16399 log.cc:824] T 322261cd38b64fb2abd5225ed992ef35 P 28eabc22c4e04f059339795b02f88ed5: Log is configured to *not* fsync() on all Append() calls
I20260619 08:59:13.952749 16399 tablet_bootstrap.cc:492] T 322261cd38b64fb2abd5225ed992ef35 P 28eabc22c4e04f059339795b02f88ed5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 08:59:13.953087 16399 tablet_bootstrap.cc:492] T 322261cd38b64fb2abd5225ed992ef35 P 28eabc22c4e04f059339795b02f88ed5: Bootstrap complete.
I20260619 08:59:13.954182 16399 ts_tablet_manager.cc:1397] T 322261cd38b64fb2abd5225ed992ef35 P 28eabc22c4e04f059339795b02f88ed5: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
Jun 19 08:59:13 dist-test-slave-h4g5 krb5kdc[15741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:59:13.955542 16399 raft_consensus.cc:348] T 322261cd38b64fb2abd5225ed992ef35 P 28eabc22c4e04f059339795b02f88ed5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32805 } } peers { permanent_uuid: "decab1e0f88a49b9922817b18178c8d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "28eabc22c4e04f059339795b02f88ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40207 } }
I20260619 08:59:13.955868 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 16357
I20260619 08:59:13.955912 16399 raft_consensus.cc:729] T 322261cd38b64fb2abd5225ed992ef35 P 28eabc22c4e04f059339795b02f88ed5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28eabc22c4e04f059339795b02f88ed5, State: Initialized, Role: FOLLOWER
I20260619 08:59:13.956125 16399 consensus_queue.cc:260] T 322261cd38b64fb2abd5225ed992ef35 P 28eabc22c4e04f059339795b02f88ed5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32805 } } peers { permanent_uuid: "decab1e0f88a49b9922817b18178c8d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "28eabc22c4e04f059339795b02f88ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40207 } }
I20260619 08:59:13.956173 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45001
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41155
--webserver_port=46081
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 08:59:13.956419 16399 ts_tablet_manager.cc:1428] T 322261cd38b64fb2abd5225ed992ef35 P 28eabc22c4e04f059339795b02f88ed5: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260619 08:59:13.959430 16509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:13.946479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.1:56243)
0619 08:59:13.948590 (+ 2111us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:13.948632 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:13.949160 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:13.949818 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:13.949822 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:13.950151 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:59:13.950436 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:13.950444 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:13.951602 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:13.951603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:13.952471 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:13.952474 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:13.953228 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:13.953230 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:13.953346 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:13.953849 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:59:13.953858 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:59:13.955218 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:59:13.957814 (+ 2596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:59:13.957822 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:59:13.957824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:59:13.958082 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:59:13.958319 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:59:13.958320 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:59:13.958321 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:59:13.958436 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:59:13.958788 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:59:13.958794 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:59:13.958947 (+ 153us) client_negotiation.cc:769] Sending connection context
0619 08:59:13.959051 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 08:59:13.959218 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2061,"thread_start_us":92,"threads_started":1}
I20260619 08:59:13.959676 16510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:13.946590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56243 (local address 127.0.98.62:41155)
0619 08:59:13.949451 (+ 2861us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:13.949459 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:13.949487 (+ 28us) server_negotiation.cc:407] Connection header received
0619 08:59:13.949594 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:13.949599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:13.949641 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:13.949761 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:13.950553 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:13.951474 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:13.952576 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:13.953084 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:13.953184 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:13.955324 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:13.955328 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:13.955328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:59:13.955339 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:13.957730 (+ 2391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:13.958161 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:13.958165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:13.958165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:13.958258 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:13.958494 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:13.958495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:13.958495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:13.958724 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:13.958796 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:13.959153 (+ 357us) server_negotiation.cc:299] Negotiation successful
0619 08:59:13.959305 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2809,"thread_start_us":100,"threads_started":1}
I20260619 08:59:13.960299 16507 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41155
I20260619 08:59:13.960402 16507 heartbeater.cc:464] Registering TS with master...
I20260619 08:59:13.960589 16507 heartbeater.cc:511] Master 127.0.98.62:41155 requested a full tablet report, sending...
I20260619 08:59:13.961460 16319 ts_manager.cc:194] Registered new tserver with Master: 28eabc22c4e04f059339795b02f88ed5 (127.0.98.1:40207)
I20260619 08:59:13.962718 16319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56243
I20260619 08:59:13.963732 16507 heartbeater.cc:503] Master 127.0.98.62:41155 was elected leader, sending a full tablet report...
Jun 19 08:59:13 dist-test-slave-h4g5 krb5kdc[15741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 08:59:13.972591 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:13.962405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53896 (local address 127.0.98.1:40207)
0619 08:59:13.962591 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:13.962596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:13.962865 (+ 269us) server_negotiation.cc:407] Connection header received
0619 08:59:13.962903 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:13.962905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:13.962919 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:13.963166 (+ 247us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:13.963766 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:13.965075 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:13.966297 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:13.966731 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:13.966806 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:13.969379 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:13.969387 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:13.969388 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:59:13.969395 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:13.971181 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:13.971652 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:13.971655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:13.971655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:13.971737 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:13.971919 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:13.971919 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:13.971920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:13.972136 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:13.972212 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:13.972380 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 08:59:13.972484 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":85,"threads_started":1}
I20260619 08:59:13.972872 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:13.962151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40207 (local address 127.0.0.1:53896)
0619 08:59:13.962660 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:13.962699 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:13.962771 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:13.963230 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:13.963235 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:13.963587 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:59:13.963674 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:13.963678 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:13.965172 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:13.965173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:13.966193 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:13.966196 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:13.966813 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:13.966820 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:13.966912 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:13.967460 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:59:13.967465 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:59:13.969276 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:59:13.971266 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:59:13.971273 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:59:13.971275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:59:13.971570 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:59:13.971799 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:59:13.971799 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:59:13.971800 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:59:13.971865 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:59:13.972188 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:59:13.972194 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:59:13.972255 (+ 61us) client_negotiation.cc:769] Sending connection context
0619 08:59:13.972341 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 08:59:13.972524 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":101,"threads_started":1}
I20260619 08:59:13.973531 16431 tablet_service.cc:1514] Processing DeleteTablet for tablet 322261cd38b64fb2abd5225ed992ef35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:58:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53896
I20260619 08:59:13.974206 16516 tablet_replica.cc:330] stopping tablet replica
I20260619 08:59:13.974323 16516 raft_consensus.cc:2229] T 322261cd38b64fb2abd5225ed992ef35 P 28eabc22c4e04f059339795b02f88ed5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:59:13.974395 16516 raft_consensus.cc:2258] T 322261cd38b64fb2abd5225ed992ef35 P 28eabc22c4e04f059339795b02f88ed5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:59:13.974701 16516 ts_tablet_manager.cc:1905] T 322261cd38b64fb2abd5225ed992ef35 P 28eabc22c4e04f059339795b02f88ed5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:59:13.977294 16516 ts_tablet_manager.cc:1918] T 322261cd38b64fb2abd5225ed992ef35 P 28eabc22c4e04f059339795b02f88ed5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:59:13.977377 16516 log.cc:1196] T 322261cd38b64fb2abd5225ed992ef35 P 28eabc22c4e04f059339795b02f88ed5: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/wal/wals/322261cd38b64fb2abd5225ed992ef35
I20260619 08:59:13.977586 16516 ts_tablet_manager.cc:1939] T 322261cd38b64fb2abd5225ed992ef35 P 28eabc22c4e04f059339795b02f88ed5: Deleting consensus metadata
I20260619 08:59:13.978157 16303 catalog_manager.cc:4841] TS 28eabc22c4e04f059339795b02f88ed5 (127.0.98.1:40207): tablet 322261cd38b64fb2abd5225ed992ef35 (table test-table [id=4b8fc71abc2d4352b488654978b03d38]) successfully deleted
W20260619 08:59:14.041769 16513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:59:14.041985 16513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:59:14.042018 16513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:59:14.043162 16513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:59:14.043201 16513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:59:14.043236 16513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 08:59:14.044704 16513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45001
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46081
--tserver_master_addrs=127.0.98.62:41155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:59:14.044950 16513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:59:14.045174 16513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:59:14.050168 16521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:14.050212 16524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:14.050213 16513 server_base.cc:1034] running on GCE node
W20260619 08:59:14.050168 16522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:14.050520 16513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:59:14.050733 16513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:59:14.051862 16513 hybrid_clock.cc:648] HybridClock initialized: now 1781859554051850 us; error 28 us; skew 500 ppm
Jun 19 08:59:14 dist-test-slave-h4g5 krb5kdc[15741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:59:14.054144 16513 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 08:59:14.054543 16513 webserver.cc:460] Webserver started at http://127.0.98.2:46081/ using document root <none> and password file <none>
I20260619 08:59:14.054754 16513 fs_manager.cc:359] Metadata directory not provided
I20260619 08:59:14.054812 16513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:59:14.056154 16513 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:59:14.056758 16531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:14.056906 16513 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:59:14.057001 16513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/wal
uuid: "decab1e0f88a49b9922817b18178c8d2"
format_stamp: "Formatted at 2026-06-19 08:58:53 on dist-test-slave-h4g5"
server_key: "77d3931545129c85429db132f12e298a"
server_key_iv: "b145e7877e8ffc94e458473bcdc12c75"
server_key_version: "encryptionkey@0"
I20260619 08:59:14.057327 16513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:59:14.083343 16513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:59:14.085572 16513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:59:14.085702 16513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:59:14.085922 16513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:59:14.086218 16513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:59:14.086251 16513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:14.086279 16513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:59:14.086295 16513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:14.091914 16513 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45001
I20260619 08:59:14.091979 16644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45001 every 8 connection(s)
I20260619 08:59:14.092312 16513 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 08:59:14 dist-test-slave-h4g5 krb5kdc[15741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:59:14.100730 16510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:14.092976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35483 (local address 127.0.98.62:41155)
0619 08:59:14.093019 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:14.093021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:14.093222 (+ 201us) server_negotiation.cc:407] Connection header received
0619 08:59:14.093722 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:14.093723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:14.093736 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:14.093792 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:14.094700 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:14.095242 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:14.096230 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:14.096440 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:14.096498 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:14.098425 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:14.098426 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:14.098427 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:59:14.098434 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:14.099575 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:14.099943 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:14.099943 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:14.099944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:14.099991 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:14.100222 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:14.100223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:14.100223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:14.100323 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:14.100371 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:14.100587 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 08:59:14.100654 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 08:59:14.100904 16647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:14.092965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.2:35483)
0619 08:59:14.093108 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:14.093144 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:14.093599 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:14.093847 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:14.093850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:14.094391 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:59:14.094592 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:14.094596 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:14.095314 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:14.095314 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:14.096157 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:14.096160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:14.096504 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:14.096504 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:14.096600 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:14.097066 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:59:14.097073 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:59:14.098345 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:59:14.099634 (+ 1289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:59:14.099640 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:59:14.099641 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:59:14.099847 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:59:14.100038 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:59:14.100038 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:59:14.100039 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:59:14.100172 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:59:14.100368 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:59:14.100371 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:59:14.100495 (+ 124us) client_negotiation.cc:769] Sending connection context
0619 08:59:14.100571 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 08:59:14.100706 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
I20260619 08:59:14.101425 16645 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41155
I20260619 08:59:14.101511 16645 heartbeater.cc:464] Registering TS with master...
I20260619 08:59:14.101655 16645 heartbeater.cc:511] Master 127.0.98.62:41155 requested a full tablet report, sending...
I20260619 08:59:14.102003 16319 ts_manager.cc:194] Registered new tserver with Master: decab1e0f88a49b9922817b18178c8d2 (127.0.98.2:45001)
I20260619 08:59:14.102401 16319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35483
I20260619 08:59:14.102486 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 16513
I20260619 08:59:14.102731 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:32805
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41155
--webserver_port=37297
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:59:14.183290 16648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:59:14.183493 16648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:59:14.183514 16648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:59:14.184698 16648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:59:14.184741 16648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:59:14.184770 16648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 08:59:14.186295 16648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:32805
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37297
--tserver_master_addrs=127.0.98.62:41155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:59:14.186504 16648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:59:14.186703 16648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:59:14.191469 16654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:14.191551 16656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:14.191565 16653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:14.191738 16648 server_base.cc:1034] running on GCE node
I20260619 08:59:14.191979 16648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:59:14.192240 16648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:59:14.193397 16648 hybrid_clock.cc:648] HybridClock initialized: now 1781859554193382 us; error 32 us; skew 500 ppm
Jun 19 08:59:14 dist-test-slave-h4g5 krb5kdc[15741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:59:14.195793 16648 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 08:59:14.196282 16648 webserver.cc:460] Webserver started at http://127.0.98.3:37297/ using document root <none> and password file <none>
I20260619 08:59:14.196483 16648 fs_manager.cc:359] Metadata directory not provided
I20260619 08:59:14.196545 16648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:59:14.197813 16648 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:59:14.198294 16663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:14.198446 16648 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:14.198521 16648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/wal
uuid: "83a3420a7ea94a53a4a2011b576f7a2d"
format_stamp: "Formatted at 2026-06-19 08:58:53 on dist-test-slave-h4g5"
server_key: "d4da8bc4f318a166b70b7508a05b0d70"
server_key_iv: "75369f20d790b1dcdda4d29d7e5537c4"
server_key_version: "encryptionkey@0"
I20260619 08:59:14.198858 16648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:59:14.211364 16648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:59:14.213254 16648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:59:14.213403 16648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:59:14.213625 16648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:59:14.213917 16648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:59:14.213949 16648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:14.213979 16648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:59:14.214002 16648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:14.219491 16648 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32805
I20260619 08:59:14.219543 16776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32805 every 8 connection(s)
I20260619 08:59:14.219863 16648 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 08:59:14 dist-test-slave-h4g5 krb5kdc[15741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:59:14.227382 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 16648
I20260619 08:59:14.228705 16510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:14.220538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46557 (local address 127.0.98.62:41155)
0619 08:59:14.220618 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:14.220620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:14.220779 (+ 159us) server_negotiation.cc:407] Connection header received
0619 08:59:14.221265 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:14.221266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:14.221277 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:14.221332 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:14.222264 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:14.222765 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:14.223799 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:14.224054 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:14.224164 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:14.226139 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:14.226141 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:14.226141 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:59:14.226148 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:14.227238 (+ 1090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:14.227683 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:14.227684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:14.227685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:14.227750 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:14.228030 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:14.228030 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:14.228031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:14.228192 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:14.228249 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:14.228574 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 08:59:14.228642 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 08:59:14.228988 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:14.220517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.3:46557)
0619 08:59:14.220658 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:14.220694 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:14.221159 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:14.221394 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:14.221398 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:14.221974 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:59:14.222170 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:14.222174 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:14.222857 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:14.222858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:14.223720 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:14.223723 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:14.224176 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:14.224177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:14.224286 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:14.224761 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:59:14.224769 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:59:14.226053 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:59:14.227309 (+ 1256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:59:14.227317 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:59:14.227319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:59:14.227615 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:59:14.227817 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:59:14.227818 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:59:14.227818 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:59:14.227971 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:59:14.228248 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:59:14.228254 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:59:14.228423 (+ 169us) client_negotiation.cc:769] Sending connection context
0619 08:59:14.228562 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 08:59:14.228731 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260619 08:59:14.229415 16777 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41155
I20260619 08:59:14.229506 16777 heartbeater.cc:464] Registering TS with master...
I20260619 08:59:14.229692 16777 heartbeater.cc:511] Master 127.0.98.62:41155 requested a full tablet report, sending...
I20260619 08:59:14.230147 16319 ts_manager.cc:194] Registered new tserver with Master: 83a3420a7ea94a53a4a2011b576f7a2d (127.0.98.3:32805)
I20260619 08:59:14.230535 16319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46557
I20260619 08:59:14.234745 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:14.227765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51722 (local address 127.0.98.62:41155)
0619 08:59:14.227907 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:14.227909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:14.227980 (+ 71us) server_negotiation.cc:407] Connection header received
0619 08:59:14.228065 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:14.228066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:14.228077 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:14.228174 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:14.228950 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:14.229597 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:14.230486 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:14.230755 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:14.230823 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:14.231981 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:14.231982 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:14.231982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:59:14.231989 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:14.233411 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:14.233797 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:14.233799 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:14.233799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:14.233859 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:14.234157 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:14.234158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:14.234158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:14.234285 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:14.234535 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:14.234574 (+ 39us) server_negotiation.cc:299] Negotiation successful
0619 08:59:14.234649 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":58,"threads_started":1}
I20260619 08:59:14.235690 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:59:14.238478 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:14.236160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51738 (local address 127.0.98.62:41155)
0619 08:59:14.236192 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:14.236193 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:14.236505 (+ 312us) server_negotiation.cc:407] Connection header received
0619 08:59:14.236541 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:14.236541 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:14.236581 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:14.236643 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:59:14.236737 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:14.237188 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:14.237877 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:14.238076 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:14.238139 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:14.238233 (+ 94us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:59:14.238275 (+ 42us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 08:59:14.238423 (+ 148us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51738: token expired
Metrics: {"server-negotiator.queue_time_us":14}
W20260619 08:59:14.238538 16780 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51738: token expired
W20260619 08:59:14.238581 16251 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:41155)
I20260619 08:59:14.243832 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:14.238814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51752 (local address 127.0.98.62:41155)
0619 08:59:14.238892 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:14.238894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:14.238901 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:59:14.238916 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:14.238917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:14.238923 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:14.238962 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:14.239344 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:14.239775 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:14.240476 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:14.240647 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:14.240688 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:14.241667 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:14.241677 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:14.241677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:59:14.241683 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:14.242877 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:14.243174 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:14.243175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:14.243175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:14.243230 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:14.243364 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:14.243365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:14.243365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:14.243459 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:14.243529 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:14.243707 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 08:59:14.243774 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 08:59:14.244926 16319 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51752:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:59:14.246474 16319 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:59:14.248966 16431 tablet_service.cc:1467] Processing CreateTablet for tablet e52d1782feab4dcbbbb5921c7005144b (DEFAULT_TABLE table=test-table [id=a4b259b85aa64f139292a8603fdf1c7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:59:14.249224 16431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e52d1782feab4dcbbbb5921c7005144b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:59:14.250744 16399 tablet_bootstrap.cc:492] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5: Bootstrap starting.
I20260619 08:59:14.251050 16399 tablet_bootstrap.cc:654] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5: Neither blocks nor log segments found. Creating new log.
I20260619 08:59:14.251573 16399 tablet_bootstrap.cc:492] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5: No bootstrap required, opened a new log
I20260619 08:59:14.251616 16399 ts_tablet_manager.cc:1397] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 08:59:14.251966 16399 raft_consensus.cc:348] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "decab1e0f88a49b9922817b18178c8d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32805 } } peers { permanent_uuid: "28eabc22c4e04f059339795b02f88ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40207 } }
I20260619 08:59:14.252032 16399 raft_consensus.cc:374] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:59:14.252106 16399 raft_consensus.cc:729] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28eabc22c4e04f059339795b02f88ed5, State: Initialized, Role: FOLLOWER
I20260619 08:59:14.252159 16399 consensus_queue.cc:260] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "decab1e0f88a49b9922817b18178c8d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32805 } } peers { permanent_uuid: "28eabc22c4e04f059339795b02f88ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40207 } }
I20260619 08:59:14.252254 16399 ts_tablet_manager.cc:1428] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 08:59:14.252851 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:14.248531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45001 (local address 127.0.0.1:38138)
0619 08:59:14.248829 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:14.248839 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:14.248894 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:14.249208 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:14.249210 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:14.249212 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:59:14.249274 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:14.249276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:14.250341 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:14.250343 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:14.251608 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:14.251611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:14.252566 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:14.252567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:14.252638 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:14.252664 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 08:59:14.252719 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 08:59:14.252764 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":32,"thread_start_us":107,"threads_started":1}
I20260619 08:59:14.252861 16785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:14.248875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38138 (local address 127.0.98.2:45001)
0619 08:59:14.249036 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:14.249040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:14.249049 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:59:14.249086 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:14.249088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:14.249102 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:14.249203 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:59:14.249344 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:14.250263 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:14.251735 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:14.252497 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:14.252567 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:14.252633 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:14.252735 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 08:59:14.252772 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":96,"threads_started":1}
I20260619 08:59:14.253327 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:14.248500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32805 (local address 127.0.0.1:42196)
0619 08:59:14.248615 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:14.248625 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:14.248657 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:14.249098 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:14.249099 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:14.249101 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:59:14.249152 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:14.249154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:14.250468 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:14.250469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:14.251656 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:14.251658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:14.253118 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:14.253120 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:14.253197 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:14.253201 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:59:14.253245 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 08:59:14.253281 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":91}
I20260619 08:59:14.253389 16784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:14.248658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42196 (local address 127.0.98.3:32805)
0619 08:59:14.248851 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:14.248856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:14.248867 (+ 11us) server_negotiation.cc:407] Connection header received
0619 08:59:14.248945 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:14.248948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:14.248964 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:14.249091 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:59:14.249372 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:14.250372 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:14.252328 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:14.253041 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:14.253111 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:14.253167 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:14.253269 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 08:59:14.253307 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":102,"threads_started":1}
I20260619 08:59:14.253795 16579 tablet_service.cc:1467] Processing CreateTablet for tablet e52d1782feab4dcbbbb5921c7005144b (DEFAULT_TABLE table=test-table [id=a4b259b85aa64f139292a8603fdf1c7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:59:14.254014 16711 tablet_service.cc:1467] Processing CreateTablet for tablet e52d1782feab4dcbbbb5921c7005144b (DEFAULT_TABLE table=test-table [id=a4b259b85aa64f139292a8603fdf1c7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:59:14.254199 16579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e52d1782feab4dcbbbb5921c7005144b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:59:14.254307 16711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e52d1782feab4dcbbbb5921c7005144b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:59:14.257954 16787 tablet_bootstrap.cc:492] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d: Bootstrap starting.
I20260619 08:59:14.258560 16787 tablet_bootstrap.cc:654] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d: Neither blocks nor log segments found. Creating new log.
I20260619 08:59:14.259020 16787 log.cc:824] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d: Log is configured to *not* fsync() on all Append() calls
I20260619 08:59:14.259322 16788 tablet_bootstrap.cc:492] T e52d1782feab4dcbbbb5921c7005144b P decab1e0f88a49b9922817b18178c8d2: Bootstrap starting.
I20260619 08:59:14.259687 16787 tablet_bootstrap.cc:492] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d: No bootstrap required, opened a new log
I20260619 08:59:14.259752 16787 ts_tablet_manager.cc:1397] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:59:14.259852 16788 tablet_bootstrap.cc:654] T e52d1782feab4dcbbbb5921c7005144b P decab1e0f88a49b9922817b18178c8d2: Neither blocks nor log segments found. Creating new log.
I20260619 08:59:14.260426 16788 log.cc:824] T e52d1782feab4dcbbbb5921c7005144b P decab1e0f88a49b9922817b18178c8d2: Log is configured to *not* fsync() on all Append() calls
I20260619 08:59:14.261106 16788 tablet_bootstrap.cc:492] T e52d1782feab4dcbbbb5921c7005144b P decab1e0f88a49b9922817b18178c8d2: No bootstrap required, opened a new log
I20260619 08:59:14.261163 16788 ts_tablet_manager.cc:1397] T e52d1782feab4dcbbbb5921c7005144b P decab1e0f88a49b9922817b18178c8d2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:59:14.261482 16787 raft_consensus.cc:348] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "decab1e0f88a49b9922817b18178c8d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32805 } } peers { permanent_uuid: "28eabc22c4e04f059339795b02f88ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40207 } }
I20260619 08:59:14.261624 16787 raft_consensus.cc:374] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:59:14.261656 16787 raft_consensus.cc:729] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83a3420a7ea94a53a4a2011b576f7a2d, State: Initialized, Role: FOLLOWER
I20260619 08:59:14.261771 16787 consensus_queue.cc:260] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "decab1e0f88a49b9922817b18178c8d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32805 } } peers { permanent_uuid: "28eabc22c4e04f059339795b02f88ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40207 } }
I20260619 08:59:14.262015 16787 ts_tablet_manager.cc:1428] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:59:14.262068 16777 heartbeater.cc:503] Master 127.0.98.62:41155 was elected leader, sending a full tablet report...
I20260619 08:59:14.262775 16788 raft_consensus.cc:348] T e52d1782feab4dcbbbb5921c7005144b P decab1e0f88a49b9922817b18178c8d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "decab1e0f88a49b9922817b18178c8d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32805 } } peers { permanent_uuid: "28eabc22c4e04f059339795b02f88ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40207 } }
I20260619 08:59:14.262953 16788 raft_consensus.cc:374] T e52d1782feab4dcbbbb5921c7005144b P decab1e0f88a49b9922817b18178c8d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:59:14.262997 16788 raft_consensus.cc:729] T e52d1782feab4dcbbbb5921c7005144b P decab1e0f88a49b9922817b18178c8d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: decab1e0f88a49b9922817b18178c8d2, State: Initialized, Role: FOLLOWER
I20260619 08:59:14.263104 16788 consensus_queue.cc:260] T e52d1782feab4dcbbbb5921c7005144b P decab1e0f88a49b9922817b18178c8d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "decab1e0f88a49b9922817b18178c8d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32805 } } peers { permanent_uuid: "28eabc22c4e04f059339795b02f88ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40207 } }
I20260619 08:59:14.263338 16788 ts_tablet_manager.cc:1428] T e52d1782feab4dcbbbb5921c7005144b P decab1e0f88a49b9922817b18178c8d2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:59:14.263394 16645 heartbeater.cc:503] Master 127.0.98.62:41155 was elected leader, sending a full tablet report...
W20260619 08:59:14.342829 16646 tablet.cc:2369] T e52d1782feab4dcbbbb5921c7005144b P decab1e0f88a49b9922817b18178c8d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 08:59:14.449692 16508 tablet.cc:2369] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 08:59:14.470456 16778 tablet.cc:2369] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:59:14.626351 16791 raft_consensus.cc:482] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:59:14.626484 16791 raft_consensus.cc:504] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "decab1e0f88a49b9922817b18178c8d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32805 } } peers { permanent_uuid: "28eabc22c4e04f059339795b02f88ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40207 } }
I20260619 08:59:14.626804 16791 leader_election.cc:288] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers decab1e0f88a49b9922817b18178c8d2 (127.0.98.2:45001), 28eabc22c4e04f059339795b02f88ed5 (127.0.98.1:40207)
I20260619 08:59:14.629647 16785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:14.627056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33579 (local address 127.0.98.2:45001)
0619 08:59:14.627126 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:14.627128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:14.627261 (+ 133us) server_negotiation.cc:407] Connection header received
0619 08:59:14.627294 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:14.627295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:14.627306 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:14.627363 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:59:14.627536 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:14.627983 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:14.628997 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:14.629342 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:14.629400 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:14.629412 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:14.629546 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 08:59:14.629581 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 08:59:14.629683 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:14.627064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41661 (local address 127.0.98.1:40207)
0619 08:59:14.627202 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:14.627204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:14.627212 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:59:14.627243 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:14.627243 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:14.627272 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:14.627346 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:59:14.627503 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:14.628021 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:14.628996 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:14.629381 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:14.629444 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:14.629521 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:14.629577 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 08:59:14.629612 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":68,"threads_started":1}
I20260619 08:59:14.629686 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:14.627030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45001 (local address 127.0.98.3:33579)
0619 08:59:14.627196 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:14.627204 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:14.627249 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:14.627381 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:14.627382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:14.627384 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:59:14.627453 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:14.627455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:14.628074 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:14.628075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:14.628908 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:14.628911 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:14.629412 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:14.629413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:14.629483 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:14.629487 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:59:14.629532 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 08:59:14.629570 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":12,"thread_start_us":100,"threads_started":1}
I20260619 08:59:14.629690 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:14.626998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40207 (local address 127.0.98.3:41661)
0619 08:59:14.627057 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:14.627069 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:14.627110 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:14.627356 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:14.627357 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:14.627359 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:59:14.627437 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:14.627439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:14.628114 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:14.628115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:14.628909 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:14.628912 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:14.629456 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:14.629457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:14.629525 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:14.629528 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:59:14.629574 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 08:59:14.629611 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":38}
I20260619 08:59:14.629987 16599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e52d1782feab4dcbbbb5921c7005144b" candidate_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decab1e0f88a49b9922817b18178c8d2" is_pre_election: true
I20260619 08:59:14.629992 16461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e52d1782feab4dcbbbb5921c7005144b" candidate_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28eabc22c4e04f059339795b02f88ed5" is_pre_election: true
I20260619 08:59:14.630134 16599 raft_consensus.cc:2454] T e52d1782feab4dcbbbb5921c7005144b P decab1e0f88a49b9922817b18178c8d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83a3420a7ea94a53a4a2011b576f7a2d in term 0.
I20260619 08:59:14.630133 16461 raft_consensus.cc:2454] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83a3420a7ea94a53a4a2011b576f7a2d in term 0.
I20260619 08:59:14.630332 16664 leader_election.cc:302] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28eabc22c4e04f059339795b02f88ed5, 83a3420a7ea94a53a4a2011b576f7a2d; no voters:
I20260619 08:59:14.630453 16791 raft_consensus.cc:2781] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:59:14.630508 16791 raft_consensus.cc:482] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:59:14.630537 16791 raft_consensus.cc:3037] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:59:14.631178 16791 raft_consensus.cc:504] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "decab1e0f88a49b9922817b18178c8d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32805 } } peers { permanent_uuid: "28eabc22c4e04f059339795b02f88ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40207 } }
I20260619 08:59:14.631290 16791 leader_election.cc:288] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d [CANDIDATE]: Term 1 election: Requested vote from peers decab1e0f88a49b9922817b18178c8d2 (127.0.98.2:45001), 28eabc22c4e04f059339795b02f88ed5 (127.0.98.1:40207)
I20260619 08:59:14.631456 16599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e52d1782feab4dcbbbb5921c7005144b" candidate_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "decab1e0f88a49b9922817b18178c8d2"
I20260619 08:59:14.631481 16461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e52d1782feab4dcbbbb5921c7005144b" candidate_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28eabc22c4e04f059339795b02f88ed5"
I20260619 08:59:14.631551 16461 raft_consensus.cc:3037] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:59:14.631541 16599 raft_consensus.cc:3037] T e52d1782feab4dcbbbb5921c7005144b P decab1e0f88a49b9922817b18178c8d2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:59:14.632359 16461 raft_consensus.cc:2454] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83a3420a7ea94a53a4a2011b576f7a2d in term 1.
I20260619 08:59:14.632385 16599 raft_consensus.cc:2454] T e52d1782feab4dcbbbb5921c7005144b P decab1e0f88a49b9922817b18178c8d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83a3420a7ea94a53a4a2011b576f7a2d in term 1.
I20260619 08:59:14.632603 16665 leader_election.cc:302] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83a3420a7ea94a53a4a2011b576f7a2d, decab1e0f88a49b9922817b18178c8d2; no voters:
I20260619 08:59:14.632732 16791 raft_consensus.cc:2781] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:59:14.632879 16791 raft_consensus.cc:686] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d [term 1 LEADER]: Becoming Leader. State: Replica: 83a3420a7ea94a53a4a2011b576f7a2d, State: Running, Role: LEADER
I20260619 08:59:14.632975 16791 consensus_queue.cc:237] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "decab1e0f88a49b9922817b18178c8d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32805 } } peers { permanent_uuid: "28eabc22c4e04f059339795b02f88ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40207 } }
I20260619 08:59:14.633769 16319 catalog_manager.cc:5495] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 83a3420a7ea94a53a4a2011b576f7a2d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "decab1e0f88a49b9922817b18178c8d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28eabc22c4e04f059339795b02f88ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40207 } health_report { overall_health: UNKNOWN } } }
I20260619 08:59:14.691900 16784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:14.689697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42210 (local address 127.0.98.3:32805)
0619 08:59:14.689757 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:14.689759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:14.689766 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:59:14.689800 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:14.689801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:14.689820 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:14.689882 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:59:14.690000 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:14.690471 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:14.691179 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:14.691366 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:14.691426 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:14.691569 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:59:14.691672 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:59:14.691742 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:14.691778 (+ 36us) server_negotiation.cc:299] Negotiation successful
0619 08:59:14.691798 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 08:59:14.693634 16461 raft_consensus.cc:1261] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5 [term 1 FOLLOWER]: Refusing update from remote peer 83a3420a7ea94a53a4a2011b576f7a2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:59:14.693634 16599 raft_consensus.cc:1261] T e52d1782feab4dcbbbb5921c7005144b P decab1e0f88a49b9922817b18178c8d2 [term 1 FOLLOWER]: Refusing update from remote peer 83a3420a7ea94a53a4a2011b576f7a2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:59:14.693962 16791 consensus_queue.cc:1035] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "decab1e0f88a49b9922817b18178c8d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:59:14.694104 16795 consensus_queue.cc:1035] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "28eabc22c4e04f059339795b02f88ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:59:14.696074 16797 mvcc.cc:205] Tried to move back new op lower bound from 7298496736022044672 to 7298496735776923648. Current Snapshot: MvccSnapshot[applied={T|T < 7298496736022044672}]
I20260619 08:59:14.696821 16799 mvcc.cc:205] Tried to move back new op lower bound from 7298496736022044672 to 7298496735776923648. Current Snapshot: MvccSnapshot[applied={T|T < 7298496736022044672}]
I20260619 08:59:14.698856 16318 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:59:14.699005 16318 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:59:14.699769 16318 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b: Sending DeleteTablet for 3 replicas of tablet e52d1782feab4dcbbbb5921c7005144b
I20260619 08:59:14.700129 16711 tablet_service.cc:1514] Processing DeleteTablet for tablet e52d1782feab4dcbbbb5921c7005144b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:59:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42196
I20260619 08:59:14.700151 16579 tablet_service.cc:1514] Processing DeleteTablet for tablet e52d1782feab4dcbbbb5921c7005144b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:59:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38138
I20260619 08:59:14.700320 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 16357
I20260619 08:59:14.700336 16431 tablet_service.cc:1514] Processing DeleteTablet for tablet e52d1782feab4dcbbbb5921c7005144b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:59:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53896
I20260619 08:59:14.700384 16809 tablet_replica.cc:330] T e52d1782feab4dcbbbb5921c7005144b P decab1e0f88a49b9922817b18178c8d2: stopping tablet replica
I20260619 08:59:14.700508 16809 raft_consensus.cc:2229] T e52d1782feab4dcbbbb5921c7005144b P decab1e0f88a49b9922817b18178c8d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:59:14.700623 16809 raft_consensus.cc:2258] T e52d1782feab4dcbbbb5921c7005144b P decab1e0f88a49b9922817b18178c8d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:59:14.702095 16809 ts_tablet_manager.cc:1905] T e52d1782feab4dcbbbb5921c7005144b P decab1e0f88a49b9922817b18178c8d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:59:14.703675 16809 ts_tablet_manager.cc:1918] T e52d1782feab4dcbbbb5921c7005144b P decab1e0f88a49b9922817b18178c8d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:59:14.703740 16809 log.cc:1196] T e52d1782feab4dcbbbb5921c7005144b P decab1e0f88a49b9922817b18178c8d2: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/wal/wals/e52d1782feab4dcbbbb5921c7005144b
I20260619 08:59:14.703962 16809 ts_tablet_manager.cc:1939] T e52d1782feab4dcbbbb5921c7005144b P decab1e0f88a49b9922817b18178c8d2: Deleting consensus metadata
I20260619 08:59:14.704380 16304 catalog_manager.cc:4841] TS decab1e0f88a49b9922817b18178c8d2 (127.0.98.2:45001): tablet e52d1782feab4dcbbbb5921c7005144b (table test-table [id=a4b259b85aa64f139292a8603fdf1c7e]) successfully deleted
W20260619 08:59:14.706957 16303 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40207 (error 108)
I20260619 08:59:14.707319 16808 tablet_replica.cc:330] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d: stopping tablet replica
I20260619 08:59:14.707427 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:14.707323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40207 (local address 127.0.0.1:53902)
0619 08:59:14.707366 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:14.707398 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 08:59:14.707525 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 16513
W20260619 08:59:14.707566 16303 catalog_manager.cc:4584] TS 28eabc22c4e04f059339795b02f88ed5 (127.0.98.1:40207): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e52d1782feab4dcbbbb5921c7005144b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40207: connect: Connection refused (error 111)
I20260619 08:59:14.712720 16808 raft_consensus.cc:2229] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:59:14.713001 16808 raft_consensus.cc:2258] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:59:14.713501 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 16648
I20260619 08:59:14.713645 16808 ts_tablet_manager.cc:1905] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:59:14.717722 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 16286
I20260619 08:59:14.717796 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:14.717678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32805 (local address 127.0.0.1:42216)
0619 08:59:14.717722 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:14.717761 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:32805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260619 08:59:34.722623 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41155
--webserver_interface=127.0.98.62
--webserver_port=40485
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:59:34.801488 16811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:59:34.801689 16811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:59:34.801710 16811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:59:34.802816 16811 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 08:59:34.802855 16811 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 08:59:34.802866 16811 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 08:59:34.802876 16811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:59:34.802888 16811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:59:34.802903 16811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:59:34.802927 16811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:59:34.802943 16811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:59:34.804389 16811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:59:34.804639 16811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:59:34.804852 16811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 08:59:34.809644 16811 server_base.cc:1034] running on GCE node
W20260619 08:59:34.809741 16817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:34.809743 16816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:34.809862 16819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:34.810134 16811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:59:34.810406 16811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:59:34.811553 16811 hybrid_clock.cc:648] HybridClock initialized: now 1781859574811537 us; error 33 us; skew 500 ppm
Jun 19 08:59:34 dist-test-slave-h4g5 krb5kdc[15741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:59:34.814152 16811 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 08:59:34.814670 16811 webserver.cc:460] Webserver started at http://127.0.98.62:40485/ using document root <none> and password file <none>
I20260619 08:59:34.814873 16811 fs_manager.cc:359] Metadata directory not provided
I20260619 08:59:34.814920 16811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:59:34.816252 16811 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:59:34.816901 16826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:34.817054 16811 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:34.817139 16811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "19abdfe959a1427f91d2c92fa822266b"
format_stamp: "Formatted at 2026-06-19 08:58:53 on dist-test-slave-h4g5"
server_key: "39e9d43f289b6845e9ce204af44cd31c"
server_key_iv: "01c7fda7edd719f6f86a5c354ff89914"
server_key_version: "encryptionkey@0"
I20260619 08:59:34.817478 16811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:59:34.823787 16811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:59:34.825629 16811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:59:34.825764 16811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:59:34.829572 16811 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41155
I20260619 08:59:34.829600 16878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41155 every 8 connection(s)
I20260619 08:59:34.829926 16811 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:59:34.831265 16879 sys_catalog.cc:263] Verifying existing consensus state
I20260619 08:59:34.831856 16879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b: Bootstrap starting.
I20260619 08:59:34.837226 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 16811
I20260619 08:59:34.837477 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40207
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41155
--webserver_port=39765
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 08:59:34.840485 16879 log.cc:824] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b: Log is configured to *not* fsync() on all Append() calls
I20260619 08:59:34.842597 16879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 08:59:34.842864 16879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b: Bootstrap complete.
I20260619 08:59:34.845012 16879 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19abdfe959a1427f91d2c92fa822266b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41155 } }
I20260619 08:59:34.845239 16879 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19abdfe959a1427f91d2c92fa822266b, State: Initialized, Role: FOLLOWER
I20260619 08:59:34.845400 16879 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19abdfe959a1427f91d2c92fa822266b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41155 } }
I20260619 08:59:34.845474 16879 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:59:34.845521 16879 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:59:34.845568 16879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [term 2 FOLLOWER]: Advancing to term 3
I20260619 08:59:34.846514 16879 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19abdfe959a1427f91d2c92fa822266b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41155 } }
I20260619 08:59:34.846608 16879 leader_election.cc:302] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19abdfe959a1427f91d2c92fa822266b; no voters:
I20260619 08:59:34.846776 16879 leader_election.cc:288] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 08:59:34.846824 16883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [term 3 FOLLOWER]: Leader election won for term 3
I20260619 08:59:34.846983 16883 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [term 3 LEADER]: Becoming Leader. State: Replica: 19abdfe959a1427f91d2c92fa822266b, State: Running, Role: LEADER
I20260619 08:59:34.847043 16879 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:59:34.847093 16883 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19abdfe959a1427f91d2c92fa822266b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41155 } }
I20260619 08:59:34.847359 16883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19abdfe959a1427f91d2c92fa822266b. Latest consensus state: current_term: 3 leader_uuid: "19abdfe959a1427f91d2c92fa822266b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19abdfe959a1427f91d2c92fa822266b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41155 } } }
I20260619 08:59:34.847523 16883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [sys.catalog]: This master's current role is: LEADER
I20260619 08:59:34.847466 16884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "19abdfe959a1427f91d2c92fa822266b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19abdfe959a1427f91d2c92fa822266b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41155 } } }
I20260619 08:59:34.847820 16884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b [sys.catalog]: This master's current role is: LEADER
I20260619 08:59:34.847939 16890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:59:34.848743 16890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:59:34.848847 16890 catalog_manager.cc:1259] Loaded cluster ID: 702f520165994b689af549353709df7b
I20260619 08:59:34.848922 16890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:59:34.850194 16890 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:59:34.850382 16890 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b: Loaded TSK: 0
I20260619 08:59:34.850431 16890 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 19abdfe959a1427f91d2c92fa822266b: Loaded TSK: 1
F20260619 08:59:34.850646 16890 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1255a609ad google::LogMessage::Fail() at ??:0
@ 0x7f1255a6388a google::LogMessage::SendToLog() at ??:0
@ 0x7f1255a604de google::LogMessage::Flush() at ??:0
@ 0x7f1255a61a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f12581cd104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f12581cad41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f125d9ba1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f125d9ba48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f125d9ba594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f125d980c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f125d98aad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f125792e0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1257923954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f12587876db start_thread at ??:0
@ 0x7f125455e61f clone at ??:0
W20260619 08:59:34.932803 16881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:59:34.932993 16881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:59:34.933014 16881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:59:34.934206 16881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:59:34.934245 16881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:59:34.934273 16881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 08:59:34.935717 16881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40207
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39765
--tserver_master_addrs=127.0.98.62:41155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:59:34.935951 16881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:59:34.936192 16881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:59:34.940932 16921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:34.940987 16881 server_base.cc:1034] running on GCE node
W20260619 08:59:34.940943 16923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:34.941084 16920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:34.941249 16881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:59:34.941464 16881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:59:34.942615 16881 hybrid_clock.cc:648] HybridClock initialized: now 1781859574942580 us; error 49 us; skew 500 ppm
Jun 19 08:59:34 dist-test-slave-h4g5 krb5kdc[15741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:59:34.944926 16881 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 08:59:34.945338 16881 webserver.cc:460] Webserver started at http://127.0.98.1:39765/ using document root <none> and password file <none>
I20260619 08:59:34.945571 16881 fs_manager.cc:359] Metadata directory not provided
I20260619 08:59:34.945623 16881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:59:34.946925 16881 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:59:34.947562 16930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:34.947722 16881 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 08:59:34.947826 16881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/wal
uuid: "28eabc22c4e04f059339795b02f88ed5"
format_stamp: "Formatted at 2026-06-19 08:58:53 on dist-test-slave-h4g5"
server_key: "263490f766468d0e26e514905f24c728"
server_key_iv: "a746b61d4dac9f74b82e5c0460297042"
server_key_version: "encryptionkey@0"
I20260619 08:59:34.948284 16881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:59:34.968063 16881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:59:34.970887 16881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:59:34.971071 16881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:59:34.971355 16881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:59:34.971904 16937 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 08:59:34.973455 16881 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 08:59:34.973522 16881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 08:59:34.973559 16881 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 08:59:34.974442 16881 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 08:59:34.974488 16881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 08:59:34.974561 16937 tablet_bootstrap.cc:492] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5: Bootstrap starting.
I20260619 08:59:34.981470 16881 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40207
I20260619 08:59:34.981673 17044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40207 every 8 connection(s)
I20260619 08:59:34.981949 16881 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
I20260619 08:59:34.982236 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 16881
I20260619 08:59:34.982537 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45001
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41155
--webserver_port=46081
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 08:59:34.984838 16937 log.cc:824] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5: Log is configured to *not* fsync() on all Append() calls
I20260619 08:59:34.989292 16937 tablet_bootstrap.cc:492] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 08:59:34.989609 16937 tablet_bootstrap.cc:492] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5: Bootstrap complete.
I20260619 08:59:34.990710 16937 ts_tablet_manager.cc:1397] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.003s
I20260619 08:59:34.992199 16937 raft_consensus.cc:348] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "decab1e0f88a49b9922817b18178c8d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32805 } } peers { permanent_uuid: "28eabc22c4e04f059339795b02f88ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40207 } }
I20260619 08:59:34.992522 16937 raft_consensus.cc:729] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28eabc22c4e04f059339795b02f88ed5, State: Initialized, Role: FOLLOWER
I20260619 08:59:34.992663 16937 consensus_queue.cc:260] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "decab1e0f88a49b9922817b18178c8d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32805 } } peers { permanent_uuid: "28eabc22c4e04f059339795b02f88ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40207 } }
I20260619 08:59:34.992956 16937 ts_tablet_manager.cc:1428] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
W20260619 08:59:35.052783 17048 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 08:59:34.984873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.1:46101)
0619 08:59:34.985063 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:34.985105 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:34.985606 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:35.052594 (+ 66988us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":103,"threads_started":1}
W20260619 08:59:35.053030 17045 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41155 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41155: Client connection negotiation failed: client connection to 127.0.98.62:41155: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 08:59:35.053387 17048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:35.053275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.1:52729)
0619 08:59:35.053319 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:35.053361 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260619 08:59:35.053685 17048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:35.053603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.1:55117)
0619 08:59:35.053642 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:35.053665 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 08:59:35.053797 17045 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 08:59:35.067260 17047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:59:35.067503 17047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:59:35.067525 17047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:59:35.069028 17047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:59:35.069083 17047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:59:35.069113 17047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 08:59:35.070660 17047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45001
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46081
--tserver_master_addrs=127.0.98.62:41155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:59:35.070894 17047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:59:35.071132 17047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:59:35.076018 17055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:35.076026 17058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:35.076031 17056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:35.076157 17047 server_base.cc:1034] running on GCE node
I20260619 08:59:35.076458 17047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:59:35.076668 17047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:59:35.077811 17047 hybrid_clock.cc:648] HybridClock initialized: now 1781859575077799 us; error 27 us; skew 500 ppm
Jun 19 08:59:35 dist-test-slave-h4g5 krb5kdc[15741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:59:35.080188 17047 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 08:59:35.080556 17047 webserver.cc:460] Webserver started at http://127.0.98.2:46081/ using document root <none> and password file <none>
I20260619 08:59:35.080737 17047 fs_manager.cc:359] Metadata directory not provided
I20260619 08:59:35.080776 17047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:59:35.081979 17047 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:59:35.082477 17065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:35.082665 17047 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:59:35.082767 17047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/wal
uuid: "decab1e0f88a49b9922817b18178c8d2"
format_stamp: "Formatted at 2026-06-19 08:58:53 on dist-test-slave-h4g5"
server_key: "77d3931545129c85429db132f12e298a"
server_key_iv: "b145e7877e8ffc94e458473bcdc12c75"
server_key_version: "encryptionkey@0"
I20260619 08:59:35.083094 17047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:59:35.109510 17047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:59:35.111625 17047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:59:35.111745 17047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:59:35.111987 17047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:59:35.112315 17047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:59:35.112349 17047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:35.112372 17047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:59:35.112394 17047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:35.117970 17047 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45001
I20260619 08:59:35.118024 17178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45001 every 8 connection(s)
I20260619 08:59:35.118340 17047 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
I20260619 08:59:35.118577 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 17047
I20260619 08:59:35.118863 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:32805
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41155
--webserver_port=37297
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 08:59:35.119467 17181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:35.119008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.2:48459)
0619 08:59:35.119181 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:35.119280 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
W20260619 08:59:35.119944 17179 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41155 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41155: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
I20260619 08:59:35.120249 17181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:35.120159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.2:43989)
0619 08:59:35.120203 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:35.120231 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 08:59:35.120522 17181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:35.120448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.2:46513)
0619 08:59:35.120484 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:35.120506 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260619 08:59:35.120609 17179 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 08:59:35.197245 17182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:59:35.197440 17182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:59:35.197463 17182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:59:35.198592 17182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:59:35.198632 17182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:59:35.198657 17182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 08:59:35.200052 17182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:32805
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37297
--tserver_master_addrs=127.0.98.62:41155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:59:35.200295 17182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:59:35.200520 17182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:59:35.205333 17188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:35.205444 17182 server_base.cc:1034] running on GCE node
W20260619 08:59:35.205334 17190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:35.205413 17187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:35.205727 17182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:59:35.205957 17182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:59:35.207093 17182 hybrid_clock.cc:648] HybridClock initialized: now 1781859575207079 us; error 30 us; skew 500 ppm
Jun 19 08:59:35 dist-test-slave-h4g5 krb5kdc[15741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:59:35.209375 17182 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 08:59:35.209749 17182 webserver.cc:460] Webserver started at http://127.0.98.3:37297/ using document root <none> and password file <none>
I20260619 08:59:35.209940 17182 fs_manager.cc:359] Metadata directory not provided
I20260619 08:59:35.209985 17182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:59:35.211208 17182 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:59:35.211902 17197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:35.212071 17182 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 08:59:35.212179 17182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/wal
uuid: "83a3420a7ea94a53a4a2011b576f7a2d"
format_stamp: "Formatted at 2026-06-19 08:58:53 on dist-test-slave-h4g5"
server_key: "d4da8bc4f318a166b70b7508a05b0d70"
server_key_iv: "75369f20d790b1dcdda4d29d7e5537c4"
server_key_version: "encryptionkey@0"
I20260619 08:59:35.212507 17182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:59:35.242508 17182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:59:35.244577 17182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:59:35.244712 17182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:59:35.244948 17182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:59:35.245349 17204 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 08:59:35.246752 17182 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 08:59:35.246791 17182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 08:59:35.246824 17182 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 08:59:35.247428 17182 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 08:59:35.247459 17182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 08:59:35.247535 17204 tablet_bootstrap.cc:492] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d: Bootstrap starting.
I20260619 08:59:35.249075 17204 log.cc:824] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d: Log is configured to *not* fsync() on all Append() calls
I20260619 08:59:35.251873 17204 tablet_bootstrap.cc:492] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 08:59:35.252175 17204 tablet_bootstrap.cc:492] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d: Bootstrap complete.
I20260619 08:59:35.252317 17204 ts_tablet_manager.cc:1397] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260619 08:59:35.253392 17182 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32805
I20260619 08:59:35.253456 17312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32805 every 8 connection(s)
I20260619 08:59:35.253715 17182 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
I20260619 08:59:35.253998 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 17182
I20260619 08:59:35.253768 17204 raft_consensus.cc:348] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "decab1e0f88a49b9922817b18178c8d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32805 } } peers { permanent_uuid: "28eabc22c4e04f059339795b02f88ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40207 } }
I20260619 08:59:35.254091 17204 raft_consensus.cc:729] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83a3420a7ea94a53a4a2011b576f7a2d, State: Initialized, Role: FOLLOWER
I20260619 08:59:35.254201 17204 consensus_queue.cc:260] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "decab1e0f88a49b9922817b18178c8d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32805 } } peers { permanent_uuid: "28eabc22c4e04f059339795b02f88ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40207 } }
I20260619 08:59:35.254467 17204 ts_tablet_manager.cc:1428] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 08:59:35.254808 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:35.254452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.3:43145)
0619 08:59:35.254606 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:35.254676 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":66,"threads_started":1}
W20260619 08:59:35.254992 17313 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41155 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41155: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
I20260619 08:59:35.255214 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:35.255128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.3:53183)
0619 08:59:35.255170 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:35.255193 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 08:59:35.255489 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:35.255425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.3:58247)
0619 08:59:35.255459 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:35.255478 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 08:59:35.255579 17313 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 08:59:36.054440 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:36.054163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.1:51549)
0619 08:59:36.054315 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:36.054369 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":59,"threads_started":1}
I20260619 08:59:36.121104 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:36.120892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.2:36345)
0619 08:59:36.121007 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:36.121044 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":55,"threads_started":1}
I20260619 08:59:36.124188 17320 raft_consensus.cc:482] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:59:36.124290 17320 raft_consensus.cc:504] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "decab1e0f88a49b9922817b18178c8d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32805 } } peers { permanent_uuid: "28eabc22c4e04f059339795b02f88ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40207 } }
I20260619 08:59:36.124622 17320 leader_election.cc:288] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers decab1e0f88a49b9922817b18178c8d2 (127.0.98.2:45001), 83a3420a7ea94a53a4a2011b576f7a2d (127.0.98.3:32805)
Jun 19 08:59:36 dist-test-slave-h4g5 krb5kdc[15741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Jun 19 08:59:36 dist-test-slave-h4g5 krb5kdc[15741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260619 08:59:36.139036 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:36.124822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32805 (local address 127.0.98.1:42875)
0619 08:59:36.124953 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:36.124962 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:36.125032 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:36.125559 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:36.125562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:36.125900 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:59:36.126092 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:36.126097 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:36.127170 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:36.127171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:36.127978 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:36.127980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:36.128680 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:36.128681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:36.128796 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:36.133658 (+ 4862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:59:36.133667 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:59:36.135462 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:59:36.137247 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:59:36.137247 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:59:36.137248 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:59:36.137628 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:59:36.137875 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:59:36.137876 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:59:36.137876 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:59:36.138022 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:59:36.138479 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:59:36.138479 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:59:36.138627 (+ 148us) client_negotiation.cc:769] Sending connection context
0619 08:59:36.138735 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 08:59:36.138927 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":62,"thread_start_us":67,"threads_started":1}
I20260619 08:59:36.139089 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:36.124985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58937 (local address 127.0.98.2:45001)
0619 08:59:36.125113 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:36.125120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:36.125145 (+ 25us) server_negotiation.cc:407] Connection header received
0619 08:59:36.125672 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:36.125674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:36.125713 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:36.125819 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:36.126192 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:36.127057 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:36.128050 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:36.128601 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:36.128694 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:36.135301 (+ 6607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:36.135305 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:36.135306 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:59:36.135316 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:36.137130 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:36.137704 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:36.137707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:36.137707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:36.137813 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:36.138081 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:36.138082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:36.138082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:36.138393 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:36.138476 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:36.138867 (+ 391us) server_negotiation.cc:299] Negotiation successful
0619 08:59:36.138992 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":53,"threads_started":1}
I20260619 08:59:36.139251 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:36.124807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45001 (local address 127.0.98.1:58937)
0619 08:59:36.124880 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:36.124892 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:36.124969 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:36.125822 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:36.125823 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:36.125995 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:59:36.126092 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:36.126098 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:36.127145 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:36.127146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:36.127975 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:36.127977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:36.128686 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:36.128687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:36.128794 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:36.133658 (+ 4864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:59:36.133667 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:59:36.135187 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:59:36.137209 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:59:36.137216 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:59:36.137218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:59:36.137628 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:59:36.137875 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:59:36.137876 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:59:36.137876 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:59:36.138022 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:59:36.138463 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:59:36.138470 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:59:36.138627 (+ 157us) client_negotiation.cc:769] Sending connection context
0619 08:59:36.138735 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 08:59:36.138999 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":112}
I20260619 08:59:36.139413 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:36.124918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42875 (local address 127.0.98.3:32805)
0619 08:59:36.125083 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:36.125094 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:36.125118 (+ 24us) server_negotiation.cc:407] Connection header received
0619 08:59:36.125243 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:36.125246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:36.125317 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:36.125495 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:36.126199 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:36.127079 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:36.128062 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:36.128594 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:36.128702 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:36.135570 (+ 6868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:36.135575 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:36.135575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:59:36.135587 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:36.137182 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:36.137704 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:36.137707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:36.137708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:36.137814 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:36.138093 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:36.138093 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:36.138094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:36.138394 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:36.138598 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:36.139206 (+ 608us) server_negotiation.cc:299] Negotiation successful
0619 08:59:36.139315 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":58,"threads_started":1}
W20260619 08:59:36.139962 16932 leader_election.cc:341] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer decab1e0f88a49b9922817b18178c8d2 (127.0.98.2:45001): Not found: Tablet not found: e52d1782feab4dcbbbb5921c7005144b
I20260619 08:59:36.139971 17267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e52d1782feab4dcbbbb5921c7005144b" candidate_uuid: "28eabc22c4e04f059339795b02f88ed5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" is_pre_election: true
I20260619 08:59:36.140133 17267 raft_consensus.cc:2454] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28eabc22c4e04f059339795b02f88ed5 in term 1.
I20260619 08:59:36.140374 16931 leader_election.cc:302] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28eabc22c4e04f059339795b02f88ed5, 83a3420a7ea94a53a4a2011b576f7a2d; no voters: decab1e0f88a49b9922817b18178c8d2
I20260619 08:59:36.140522 17320 raft_consensus.cc:2781] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 08:59:36.140651 17320 raft_consensus.cc:482] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:59:36.140707 17320 raft_consensus.cc:3037] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5 [term 1 FOLLOWER]: Advancing to term 2
I20260619 08:59:36.141853 17320 raft_consensus.cc:504] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "decab1e0f88a49b9922817b18178c8d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32805 } } peers { permanent_uuid: "28eabc22c4e04f059339795b02f88ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40207 } }
I20260619 08:59:36.141987 17320 leader_election.cc:288] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5 [CANDIDATE]: Term 2 election: Requested vote from peers decab1e0f88a49b9922817b18178c8d2 (127.0.98.2:45001), 83a3420a7ea94a53a4a2011b576f7a2d (127.0.98.3:32805)
I20260619 08:59:36.142164 17267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e52d1782feab4dcbbbb5921c7005144b" candidate_uuid: "28eabc22c4e04f059339795b02f88ed5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "83a3420a7ea94a53a4a2011b576f7a2d"
W20260619 08:59:36.142225 16932 leader_election.cc:341] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer decab1e0f88a49b9922817b18178c8d2 (127.0.98.2:45001): Not found: Tablet not found: e52d1782feab4dcbbbb5921c7005144b
I20260619 08:59:36.142248 17267 raft_consensus.cc:3037] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d [term 1 FOLLOWER]: Advancing to term 2
I20260619 08:59:36.143589 17267 raft_consensus.cc:2454] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28eabc22c4e04f059339795b02f88ed5 in term 2.
I20260619 08:59:36.143746 16931 leader_election.cc:302] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28eabc22c4e04f059339795b02f88ed5, 83a3420a7ea94a53a4a2011b576f7a2d; no voters: decab1e0f88a49b9922817b18178c8d2
I20260619 08:59:36.143829 17320 raft_consensus.cc:2781] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 08:59:36.143942 17320 raft_consensus.cc:686] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5 [term 2 LEADER]: Becoming Leader. State: Replica: 28eabc22c4e04f059339795b02f88ed5, State: Running, Role: LEADER
I20260619 08:59:36.144017 17320 consensus_queue.cc:237] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "decab1e0f88a49b9922817b18178c8d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32805 } } peers { permanent_uuid: "28eabc22c4e04f059339795b02f88ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40207 } }
I20260619 08:59:36.144232 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:36.144148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.1:50273)
0619 08:59:36.144179 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:36.144205 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260619 08:59:36.256198 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:36.255926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.3:55787)
0619 08:59:36.256111 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:36.256157 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":72,"threads_started":1}
I20260619 08:59:36.645316 17267 raft_consensus.cc:1261] T e52d1782feab4dcbbbb5921c7005144b P 83a3420a7ea94a53a4a2011b576f7a2d [term 2 FOLLOWER]: Refusing update from remote peer 28eabc22c4e04f059339795b02f88ed5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260619 08:59:36.645720 17327 consensus_queue.cc:1035] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260619 08:59:36.646567 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:36.646441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.3:33251)
0619 08:59:36.646490 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:36.646530 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260619 08:59:36.647503 16932 consensus_peers.cc:487] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5 -> Peer decab1e0f88a49b9922817b18178c8d2 (127.0.98.2:45001): Couldn't send request to peer decab1e0f88a49b9922817b18178c8d2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e52d1782feab4dcbbbb5921c7005144b. This is attempt 1: this message will repeat every 5th retry.
I20260619 08:59:36.647580 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:36.647325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.1:58743)
0619 08:59:36.647503 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:36.647540 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":81,"threads_started":1}
I20260619 08:59:36.647629 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:36.647528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.3:57245)
0619 08:59:36.647566 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:36.647598 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260619 08:59:37.121778 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:37.121557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.2:42543)
0619 08:59:37.121692 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:37.121734 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
I20260619 08:59:37.153972 17337 ts_tablet_manager.cc:927] T e52d1782feab4dcbbbb5921c7005144b P decab1e0f88a49b9922817b18178c8d2: Initiating tablet copy from peer 28eabc22c4e04f059339795b02f88ed5 (127.0.98.1:40207)
I20260619 08:59:37.154203 17337 tablet_copy_client.cc:316] T e52d1782feab4dcbbbb5921c7005144b P decab1e0f88a49b9922817b18178c8d2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.1:40207
Jun 19 08:59:37 dist-test-slave-h4g5 krb5kdc[15741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 08:59:37.161773 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:37.154547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41883 (local address 127.0.98.1:40207)
0619 08:59:37.154653 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:37.154658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:37.154666 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:59:37.154690 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:37.154691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:37.154703 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:37.154765 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:37.155298 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:37.156011 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:37.156832 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:37.157141 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:37.157197 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:37.159237 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:37.159244 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:37.159244 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:59:37.159250 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:37.160681 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:37.161073 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:37.161075 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:37.161075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:37.161125 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:37.161285 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:37.161286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:37.161286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:37.161413 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:37.161469 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:37.161639 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 08:59:37.161703 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":53,"threads_started":1}
I20260619 08:59:37.161790 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:37.154450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40207 (local address 127.0.98.2:41883)
0619 08:59:37.154503 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:37.154526 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:37.154607 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:37.154847 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:37.154854 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:37.155134 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:59:37.155213 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:37.155215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:37.156136 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:37.156138 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:37.156752 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:37.156754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:37.157199 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:37.157200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:37.157248 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:37.157733 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:59:37.157737 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:59:37.159145 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:59:37.160781 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:59:37.160788 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:59:37.160790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:59:37.161002 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:59:37.161178 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:59:37.161178 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:59:37.161178 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:59:37.161238 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:59:37.161492 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:59:37.161497 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:59:37.161545 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 08:59:37.161614 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 08:59:37.161709 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 08:59:37.162199 17019 tablet_copy_service.cc:140] P 28eabc22c4e04f059339795b02f88ed5: Received BeginTabletCopySession request for tablet e52d1782feab4dcbbbb5921c7005144b from peer decab1e0f88a49b9922817b18178c8d2 ({username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41883)
I20260619 08:59:37.162273 17019 tablet_copy_service.cc:161] P 28eabc22c4e04f059339795b02f88ed5: Beginning new tablet copy session on tablet e52d1782feab4dcbbbb5921c7005144b from peer decab1e0f88a49b9922817b18178c8d2 at {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41883: session id = decab1e0f88a49b9922817b18178c8d2-e52d1782feab4dcbbbb5921c7005144b
I20260619 08:59:37.162729 17019 tablet_copy_source_session.cc:205] T e52d1782feab4dcbbbb5921c7005144b P 28eabc22c4e04f059339795b02f88ed5: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 08:59:37.163625 17337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e52d1782feab4dcbbbb5921c7005144b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:59:37.166292 17337 tablet_copy_client.cc:791] T e52d1782feab4dcbbbb5921c7005144b P decab1e0f88a49b9922817b18178c8d2: tablet copy: Starting download of 0 data blocks...
I20260619 08:59:37.166419 17337 tablet_copy_client.cc:655] T e52d1782feab4dcbbbb5921c7005144b P decab1e0f88a49b9922817b18178c8d2: tablet copy: Starting download of 1 WAL segments...
I20260619 08:59:37.167217 17337 tablet_copy_client.cc:531] T e52d1782feab4dcbbbb5921c7005144b P decab1e0f88a49b9922817b18178c8d2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 08:59:37.167819 17337 tablet_bootstrap.cc:492] T e52d1782feab4dcbbbb5921c7005144b P decab1e0f88a49b9922817b18178c8d2: Bootstrap starting.
I20260619 08:59:37.169054 17337 log.cc:824] T e52d1782feab4dcbbbb5921c7005144b P decab1e0f88a49b9922817b18178c8d2: Log is configured to *not* fsync() on all Append() calls
I20260619 08:59:37.171103 17337 tablet_bootstrap.cc:492] T e52d1782feab4dcbbbb5921c7005144b P decab1e0f88a49b9922817b18178c8d2: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 08:59:37.171308 17337 tablet_bootstrap.cc:492] T e52d1782feab4dcbbbb5921c7005144b P decab1e0f88a49b9922817b18178c8d2: Bootstrap complete.
I20260619 08:59:37.171415 17337 ts_tablet_manager.cc:1397] T e52d1782feab4dcbbbb5921c7005144b P decab1e0f88a49b9922817b18178c8d2: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260619 08:59:37.172417 17337 raft_consensus.cc:348] T e52d1782feab4dcbbbb5921c7005144b P decab1e0f88a49b9922817b18178c8d2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "decab1e0f88a49b9922817b18178c8d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32805 } } peers { permanent_uuid: "28eabc22c4e04f059339795b02f88ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40207 } }
I20260619 08:59:37.172552 17337 raft_consensus.cc:729] T e52d1782feab4dcbbbb5921c7005144b P decab1e0f88a49b9922817b18178c8d2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: decab1e0f88a49b9922817b18178c8d2, State: Initialized, Role: FOLLOWER
I20260619 08:59:37.172655 17337 consensus_queue.cc:260] T e52d1782feab4dcbbbb5921c7005144b P decab1e0f88a49b9922817b18178c8d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "decab1e0f88a49b9922817b18178c8d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45001 } } peers { permanent_uuid: "83a3420a7ea94a53a4a2011b576f7a2d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32805 } } peers { permanent_uuid: "28eabc22c4e04f059339795b02f88ed5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40207 } }
I20260619 08:59:37.172891 17337 ts_tablet_manager.cc:1428] T e52d1782feab4dcbbbb5921c7005144b P decab1e0f88a49b9922817b18178c8d2: Time spent starting tablet: real 0.001s user 0.002s sys 0.001s
I20260619 08:59:37.173146 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:37.173058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.2:33053)
0619 08:59:37.173088 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:37.173115 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260619 08:59:37.173147 17019 tablet_copy_service.cc:342] P 28eabc22c4e04f059339795b02f88ed5: Request end of tablet copy session decab1e0f88a49b9922817b18178c8d2-e52d1782feab4dcbbbb5921c7005144b received from {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41883
I20260619 08:59:37.173194 17019 tablet_copy_service.cc:433] P 28eabc22c4e04f059339795b02f88ed5: ending tablet copy session decab1e0f88a49b9922817b18178c8d2-e52d1782feab4dcbbbb5921c7005144b on tablet e52d1782feab4dcbbbb5921c7005144b with peer decab1e0f88a49b9922817b18178c8d2
I20260619 08:59:37.173473 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:37.173380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.2:50863)
0619 08:59:37.173414 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:37.173446 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260619 08:59:37.648216 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:37.647984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.1:58521)
0619 08:59:37.648134 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:37.648171 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":52,"threads_started":1}
I20260619 08:59:37.648216 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:37.647989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.3:53925)
0619 08:59:37.648134 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:37.648172 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":75,"threads_started":1}
I20260619 08:59:37.704579 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:37.704465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.1:56095)
0619 08:59:37.704508 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:37.704540 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 08:59:37.704597 17133 raft_consensus.cc:1203] T e52d1782feab4dcbbbb5921c7005144b P decab1e0f88a49b9922817b18178c8d2 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260619 08:59:37.705099 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:37.704945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.2:36389)
0619 08:59:37.705034 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:37.705057 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":43,"threads_started":1}
I20260619 08:59:38.648871 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:38.648646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.3:54729)
0619 08:59:38.648775 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:38.648823 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
I20260619 08:59:38.705281 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:38.705031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.1:43453)
0619 08:59:38.705179 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:38.705235 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":63,"threads_started":1}
I20260619 08:59:38.705704 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:38.705497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.2:42757)
0619 08:59:38.705600 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:38.705634 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":51,"threads_started":1}
I20260619 08:59:39.649555 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:39.649335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.3:50823)
0619 08:59:39.649471 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:39.649507 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":70,"threads_started":1}
I20260619 08:59:39.705909 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:39.705703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.1:53059)
0619 08:59:39.705814 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:39.705857 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":60,"threads_started":1}
I20260619 08:59:39.706264 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:39.706095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.2:42669)
0619 08:59:39.706191 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:39.706226 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":43,"threads_started":1}
I20260619 08:59:40.650405 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:40.650187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.3:56481)
0619 08:59:40.650310 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:40.650357 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
I20260619 08:59:40.706868 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:40.706647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.1:59931)
0619 08:59:40.706779 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:40.706823 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260619 08:59:40.706876 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:40.706700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.2:49397)
0619 08:59:40.706805 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:40.706837 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":41,"threads_started":1}
I20260619 08:59:40.707250 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:40.707169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.1:49555)
0619 08:59:40.707211 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:40.707234 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 08:59:41.651103 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:41.650854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.3:45069)
0619 08:59:41.650998 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:41.651041 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":72,"threads_started":1}
I20260619 08:59:41.707615 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:41.707343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.2:53469)
0619 08:59:41.707518 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:41.707568 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":65,"threads_started":1}
I20260619 08:59:41.707831 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:41.707635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.1:44717)
0619 08:59:41.707742 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:41.707774 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":42,"threads_started":1}
I20260619 08:59:42.651758 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:42.651531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.3:44913)
0619 08:59:42.651671 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:42.651703 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260619 08:59:42.708446 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:42.708236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.1:38903)
0619 08:59:42.708373 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:42.708403 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":54,"threads_started":1}
I20260619 08:59:42.708436 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:42.708205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.2:42391)
0619 08:59:42.708342 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:42.708382 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
I20260619 08:59:43.652550 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:43.652321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.3:40559)
0619 08:59:43.652453 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:43.652494 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
I20260619 08:59:43.709101 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:43.708912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.1:60403)
0619 08:59:43.709033 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:43.709061 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260619 08:59:43.709100 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:43.708887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.2:57957)
0619 08:59:43.709019 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:43.709056 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
I20260619 08:59:44.653116 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:44.652920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.3:58403)
0619 08:59:44.653033 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:44.653068 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":59,"threads_started":1}
I20260619 08:59:44.709853 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:44.709619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.1:40405)
0619 08:59:44.709774 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:44.709807 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":73,"threads_started":1}
I20260619 08:59:44.709854 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:44.709619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.2:46295)
0619 08:59:44.709759 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:44.709806 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
I20260619 08:59:45.653762 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:45.653567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.3:51483)
0619 08:59:45.653686 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:45.653716 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
I20260619 08:59:45.710580 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:45.710360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.2:35183)
0619 08:59:45.710496 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:45.710536 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":80,"threads_started":1}
I20260619 08:59:45.710593 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:45.710360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.1:51509)
0619 08:59:45.710523 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:45.710553 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":82,"threads_started":1}
I20260619 08:59:46.654400 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:46.654161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.3:37467)
0619 08:59:46.654294 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:46.654336 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
I20260619 08:59:46.711277 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:46.711060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.2:51089)
0619 08:59:46.711194 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:46.711225 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
I20260619 08:59:46.711277 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:46.711061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.1:43081)
0619 08:59:46.711194 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:46.711226 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":68,"threads_started":1}
I20260619 08:59:47.655036 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:47.654819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.3:42417)
0619 08:59:47.654946 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:47.654989 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260619 08:59:47.711966 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:47.711767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.1:33971)
0619 08:59:47.711885 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:47.711918 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":55,"threads_started":1}
I20260619 08:59:47.711974 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:47.711757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.2:34927)
0619 08:59:47.711900 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:47.711934 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":58,"threads_started":1}
I20260619 08:59:48.655869 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:48.655569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.3:50999)
0619 08:59:48.655728 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:48.655815 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":66,"threads_started":1}
I20260619 08:59:48.712648 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:48.712425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.1:45387)
0619 08:59:48.712565 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:48.712600 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":55,"threads_started":1}
I20260619 08:59:48.712647 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:48.712404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.2:55601)
0619 08:59:48.712573 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:48.712606 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":82,"threads_started":1}
I20260619 08:59:49.656627 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:49.656361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.3:59141)
0619 08:59:49.656526 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:49.656575 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":76,"threads_started":1}
I20260619 08:59:49.713322 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:49.713112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.1:39253)
0619 08:59:49.713254 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:49.713281 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":75,"threads_started":1}
I20260619 08:59:49.713322 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:49.713098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41155 (local address 127.0.98.2:33753)
0619 08:59:49.713247 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:49.713278 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
I20260619 08:59:50.258949 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 16881
I20260619 08:59:50.263154 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 17047
I20260619 08:59:50.267093 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 17182
2026-06-19T08:59:50Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58414 ms)
[----------] 30 tests from SecurityITest (228732 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:59:50 dist-test-slave-h4g5 krb5kdc[17387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:59:50 dist-test-slave-h4g5 krb5kdc[17387](info): set up 2 sockets
Jun 19 08:59:50 dist-test-slave-h4g5 krb5kdc[17387](info): commencing operation
krb5kdc: starting...
W20260619 08:59:51.227634 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 08:59:51 dist-test-slave-h4g5 krb5kdc[17387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:59:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:59:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 08:59:51.343482 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35937
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35937
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:59:51.423575 17400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:59:51.423805 17400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:59:51.423827 17400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:59:51.425041 17400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:59:51.425086 17400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:59:51.425098 17400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:59:51.425110 17400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:59:51.425120 17400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:59:51.426618 17400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:59:51.426884 17400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:59:51.427096 17400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:59:51.431692 17406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:51.431733 17408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:51.431733 17405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:51.431833 17400 server_base.cc:1034] running on GCE node
I20260619 08:59:51.432188 17400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:59:51.432489 17400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:59:51.433651 17400 hybrid_clock.cc:648] HybridClock initialized: now 1781859591433627 us; error 37 us; skew 500 ppm
Jun 19 08:59:51 dist-test-slave-h4g5 krb5kdc[17387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:59:51.435956 17400 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 08:59:51.436379 17400 webserver.cc:460] Webserver started at http://127.0.98.62:40181/ using document root <none> and password file <none>
I20260619 08:59:51.436563 17400 fs_manager.cc:359] Metadata directory not provided
I20260619 08:59:51.436602 17400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:59:51.436686 17400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:59:51.437682 17400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "711bdd72c206433996e5fb9a844fde7f"
format_stamp: "Formatted at 2026-06-19 08:59:51 on dist-test-slave-h4g5"
server_key: "26c2304a19629036d9582fb98ac8d6c2"
server_key_iv: "d22d2d95d65e35303fbf9e6e13750b02"
server_key_version: "encryptionkey@0"
I20260619 08:59:51.437984 17400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "711bdd72c206433996e5fb9a844fde7f"
format_stamp: "Formatted at 2026-06-19 08:59:51 on dist-test-slave-h4g5"
server_key: "26c2304a19629036d9582fb98ac8d6c2"
server_key_iv: "d22d2d95d65e35303fbf9e6e13750b02"
server_key_version: "encryptionkey@0"
I20260619 08:59:51.439105 17400 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:59:51.439750 17415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:51.439889 17400 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 08:59:51.439965 17400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "711bdd72c206433996e5fb9a844fde7f"
format_stamp: "Formatted at 2026-06-19 08:59:51 on dist-test-slave-h4g5"
server_key: "26c2304a19629036d9582fb98ac8d6c2"
server_key_iv: "d22d2d95d65e35303fbf9e6e13750b02"
server_key_version: "encryptionkey@0"
I20260619 08:59:51.440018 17400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:59:51.475706 17400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:59:51.477869 17400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:59:51.477982 17400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:59:51.481634 17400 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35937
I20260619 08:59:51.481696 17467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35937 every 8 connection(s)
I20260619 08:59:51.482002 17400 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:59:51.482851 17468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:59:51.485065 17468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 711bdd72c206433996e5fb9a844fde7f: Bootstrap starting.
I20260619 08:59:51.485591 17468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 711bdd72c206433996e5fb9a844fde7f: Neither blocks nor log segments found. Creating new log.
I20260619 08:59:51.485845 17468 log.cc:824] T 00000000000000000000000000000000 P 711bdd72c206433996e5fb9a844fde7f: Log is configured to *not* fsync() on all Append() calls
I20260619 08:59:51.486505 17468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 711bdd72c206433996e5fb9a844fde7f: No bootstrap required, opened a new log
I20260619 08:59:51.488062 17468 raft_consensus.cc:348] T 00000000000000000000000000000000 P 711bdd72c206433996e5fb9a844fde7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "711bdd72c206433996e5fb9a844fde7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35937 } }
I20260619 08:59:51.488243 17468 raft_consensus.cc:374] T 00000000000000000000000000000000 P 711bdd72c206433996e5fb9a844fde7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:59:51.488271 17468 raft_consensus.cc:729] T 00000000000000000000000000000000 P 711bdd72c206433996e5fb9a844fde7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 711bdd72c206433996e5fb9a844fde7f, State: Initialized, Role: FOLLOWER
I20260619 08:59:51.488384 17468 consensus_queue.cc:260] T 00000000000000000000000000000000 P 711bdd72c206433996e5fb9a844fde7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "711bdd72c206433996e5fb9a844fde7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35937 } }
I20260619 08:59:51.488488 17468 raft_consensus.cc:388] T 00000000000000000000000000000000 P 711bdd72c206433996e5fb9a844fde7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:59:51.488520 17468 raft_consensus.cc:482] T 00000000000000000000000000000000 P 711bdd72c206433996e5fb9a844fde7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:59:51.488570 17468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 711bdd72c206433996e5fb9a844fde7f [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:59:51.488564 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 17400
I20260619 08:59:51.488654 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 08:59:51.488838 392 external_mini_cluster.cc:1428] Setting key 0ce81a603348ba1cf3720593a0e2fce8
I20260619 08:59:51.489208 17468 raft_consensus.cc:504] T 00000000000000000000000000000000 P 711bdd72c206433996e5fb9a844fde7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "711bdd72c206433996e5fb9a844fde7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35937 } }
I20260619 08:59:51.489323 17468 leader_election.cc:302] T 00000000000000000000000000000000 P 711bdd72c206433996e5fb9a844fde7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 711bdd72c206433996e5fb9a844fde7f; no voters:
I20260619 08:59:51.489490 17468 leader_election.cc:288] T 00000000000000000000000000000000 P 711bdd72c206433996e5fb9a844fde7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:59:51.489513 17473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 711bdd72c206433996e5fb9a844fde7f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:59:51.489706 17473 raft_consensus.cc:686] T 00000000000000000000000000000000 P 711bdd72c206433996e5fb9a844fde7f [term 1 LEADER]: Becoming Leader. State: Replica: 711bdd72c206433996e5fb9a844fde7f, State: Running, Role: LEADER
I20260619 08:59:51.489792 17473 consensus_queue.cc:237] T 00000000000000000000000000000000 P 711bdd72c206433996e5fb9a844fde7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "711bdd72c206433996e5fb9a844fde7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35937 } }
I20260619 08:59:51.489840 17468 sys_catalog.cc:564] T 00000000000000000000000000000000 P 711bdd72c206433996e5fb9a844fde7f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:59:51.490098 17473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 711bdd72c206433996e5fb9a844fde7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "711bdd72c206433996e5fb9a844fde7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "711bdd72c206433996e5fb9a844fde7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35937 } } }
I20260619 08:59:51.490252 17473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 711bdd72c206433996e5fb9a844fde7f [sys.catalog]: This master's current role is: LEADER
I20260619 08:59:51.490434 17473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 711bdd72c206433996e5fb9a844fde7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 711bdd72c206433996e5fb9a844fde7f. Latest consensus state: current_term: 1 leader_uuid: "711bdd72c206433996e5fb9a844fde7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "711bdd72c206433996e5fb9a844fde7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35937 } } }
I20260619 08:59:51.490504 17473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 711bdd72c206433996e5fb9a844fde7f [sys.catalog]: This master's current role is: LEADER
I20260619 08:59:51.490797 17481 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:59:51.491268 17481 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:59:51.493062 17481 catalog_manager.cc:1347] Generated new cluster ID: ea32a867b29e409c94277480e7930a2f
I20260619 08:59:51.493115 17481 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 08:59:51 dist-test-slave-h4g5 krb5kdc[17387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:59:51.500927 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:51.489297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44206 (local address 127.0.98.62:35937)
0619 08:59:51.489490 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:51.489501 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:51.489535 (+ 34us) server_negotiation.cc:407] Connection header received
0619 08:59:51.489655 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:51.489663 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:51.489727 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:51.489918 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:51.490761 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:51.492197 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:51.493212 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:51.493816 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:51.493907 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:51.496616 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:51.496621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:51.496621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:59:51.496634 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:51.499036 (+ 2402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:51.499419 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:51.499423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:51.499423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:51.499548 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:51.499733 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:51.499734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:51.499734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:51.500021 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:51.500170 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:51.500579 (+ 409us) server_negotiation.cc:299] Negotiation successful
0619 08:59:51.500712 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":116,"threads_started":1}
I20260619 08:59:51.509574 17481 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:59:51.510097 17481 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:59:51.513082 17481 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 711bdd72c206433996e5fb9a844fde7f: Generated new TSK 0
I20260619 08:59:51.513273 17481 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 08:59:51.538780 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35937
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:59:51.618539 17494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:59:51.618742 17494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:59:51.618767 17494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:59:51.619943 17494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:59:51.619992 17494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:59:51.620023 17494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 08:59:51.621585 17494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:59:51.621826 17494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:59:51.622043 17494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:59:51.626780 17502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:51.626854 17499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:51.626989 17494 server_base.cc:1034] running on GCE node
W20260619 08:59:51.626880 17500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:51.627300 17494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:59:51.627499 17494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:59:51.628633 17494 hybrid_clock.cc:648] HybridClock initialized: now 1781859591628620 us; error 28 us; skew 500 ppm
Jun 19 08:59:51 dist-test-slave-h4g5 krb5kdc[17387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:59:51.631014 17494 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 08:59:51.631445 17494 webserver.cc:460] Webserver started at http://127.0.98.1:40451/ using document root <none> and password file <none>
I20260619 08:59:51.631649 17494 fs_manager.cc:359] Metadata directory not provided
I20260619 08:59:51.631695 17494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:59:51.631817 17494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:59:51.632849 17494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-0/data/instance:
uuid: "53063539bae848788059f09418c88336"
format_stamp: "Formatted at 2026-06-19 08:59:51 on dist-test-slave-h4g5"
server_key: "60f3d0416819185e17dc8925f321c859"
server_key_iv: "aba509305d1376747c62f480603b354e"
server_key_version: "encryptionkey@0"
I20260619 08:59:51.633189 17494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-0/wal/instance:
uuid: "53063539bae848788059f09418c88336"
format_stamp: "Formatted at 2026-06-19 08:59:51 on dist-test-slave-h4g5"
server_key: "60f3d0416819185e17dc8925f321c859"
server_key_iv: "aba509305d1376747c62f480603b354e"
server_key_version: "encryptionkey@0"
I20260619 08:59:51.634347 17494 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 08:59:51.634938 17509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:51.635082 17494 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 08:59:51.635159 17494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-0/wal
uuid: "53063539bae848788059f09418c88336"
format_stamp: "Formatted at 2026-06-19 08:59:51 on dist-test-slave-h4g5"
server_key: "60f3d0416819185e17dc8925f321c859"
server_key_iv: "aba509305d1376747c62f480603b354e"
server_key_version: "encryptionkey@0"
I20260619 08:59:51.635226 17494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:59:51.647001 17494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:59:51.648829 17494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:59:51.648960 17494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:59:51.649164 17494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:59:51.649451 17494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:59:51.649484 17494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:51.649513 17494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:59:51.649529 17494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:51.655009 17494 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35319
I20260619 08:59:51.655048 17622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35319 every 8 connection(s)
I20260619 08:59:51.655347 17494 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 08:59:51 dist-test-slave-h4g5 krb5kdc[17387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:59:51.663692 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 17494
I20260619 08:59:51.663797 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-0/wal/instance
I20260619 08:59:51.663978 392 external_mini_cluster.cc:1428] Setting key 4ad9fa6b423332743df6a30fd90be273
I20260619 08:59:51.664377 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:51.656018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60093 (local address 127.0.98.62:35937)
0619 08:59:51.656113 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:51.656115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:51.656279 (+ 164us) server_negotiation.cc:407] Connection header received
0619 08:59:51.657181 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:51.657182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:51.657193 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:51.657260 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:51.657864 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:51.658395 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:51.659397 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:51.659597 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:51.659648 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:51.661712 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:51.661714 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:51.661714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:59:51.661720 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:51.662972 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:51.663365 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:51.663367 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:51.663367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:51.663439 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:51.663703 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:51.663703 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:51.663704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:51.663827 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:51.663890 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:51.664194 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 08:59:51.664289 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 08:59:51.664533 17625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:51.655973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35937 (local address 127.0.98.1:60093)
0619 08:59:51.656154 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:51.656198 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:51.657085 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:51.657329 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:51.657332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:51.657573 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:59:51.657779 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:51.657783 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:51.658472 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:51.658473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:51.659321 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:51.659323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:51.659686 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:51.659687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:51.659819 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:51.660332 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:59:51.660341 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:59:51.661627 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:59:51.663047 (+ 1420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:59:51.663053 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:59:51.663055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:59:51.663258 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:59:51.663503 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:59:51.663503 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:59:51.663504 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:59:51.663639 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:59:51.663884 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:59:51.663892 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:59:51.664032 (+ 140us) client_negotiation.cc:769] Sending connection context
0619 08:59:51.664158 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 08:59:51.664316 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":81,"threads_started":1}
I20260619 08:59:51.664997 17623 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35937
I20260619 08:59:51.665089 17623 heartbeater.cc:464] Registering TS with master...
I20260619 08:59:51.665254 17623 heartbeater.cc:511] Master 127.0.98.62:35937 requested a full tablet report, sending...
I20260619 08:59:51.665716 17432 ts_manager.cc:194] Registered new tserver with Master: 53063539bae848788059f09418c88336 (127.0.98.1:35319)
I20260619 08:59:51.666590 17432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60093
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 08:59:51.687036 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35937
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:59:51.767606 17628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:59:51.767839 17628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:59:51.767867 17628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:59:51.769178 17628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:59:51.769232 17628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:59:51.769260 17628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 08:59:51.770771 17628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:59:51.771016 17628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:59:51.771251 17628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:59:51.776002 17633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:51.776005 17634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:51.776014 17636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:51.776221 17628 server_base.cc:1034] running on GCE node
I20260619 08:59:51.776413 17628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:59:51.776626 17628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:59:51.777761 17628 hybrid_clock.cc:648] HybridClock initialized: now 1781859591777743 us; error 33 us; skew 500 ppm
Jun 19 08:59:51 dist-test-slave-h4g5 krb5kdc[17387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:59:51.780030 17628 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 08:59:51.780441 17628 webserver.cc:460] Webserver started at http://127.0.98.2:34275/ using document root <none> and password file <none>
I20260619 08:59:51.780630 17628 fs_manager.cc:359] Metadata directory not provided
I20260619 08:59:51.780676 17628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:59:51.780810 17628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:59:51.781852 17628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-1/data/instance:
uuid: "8ea59a825de24fe596e72b871f7f23f2"
format_stamp: "Formatted at 2026-06-19 08:59:51 on dist-test-slave-h4g5"
server_key: "afa34c5a1be182870d86b0987d839849"
server_key_iv: "eddf4b0d29ec4618f1d96b5a85454e25"
server_key_version: "encryptionkey@0"
I20260619 08:59:51.782169 17628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8ea59a825de24fe596e72b871f7f23f2"
format_stamp: "Formatted at 2026-06-19 08:59:51 on dist-test-slave-h4g5"
server_key: "afa34c5a1be182870d86b0987d839849"
server_key_iv: "eddf4b0d29ec4618f1d96b5a85454e25"
server_key_version: "encryptionkey@0"
I20260619 08:59:51.783353 17628 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:59:51.783963 17643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:51.784137 17628 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:51.784216 17628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-1/wal
uuid: "8ea59a825de24fe596e72b871f7f23f2"
format_stamp: "Formatted at 2026-06-19 08:59:51 on dist-test-slave-h4g5"
server_key: "afa34c5a1be182870d86b0987d839849"
server_key_iv: "eddf4b0d29ec4618f1d96b5a85454e25"
server_key_version: "encryptionkey@0"
I20260619 08:59:51.784281 17628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:59:51.796748 17628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:59:51.798524 17628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:59:51.798653 17628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:59:51.798873 17628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:59:51.799170 17628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:59:51.799202 17628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:51.799232 17628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:59:51.799257 17628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:51.804720 17628 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35837
I20260619 08:59:51.804792 17756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35837 every 8 connection(s)
I20260619 08:59:51.805061 17628 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 08:59:51 dist-test-slave-h4g5 krb5kdc[17387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:59:51.811646 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 17628
I20260619 08:59:51.811741 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-1/wal/instance
I20260619 08:59:51.811957 392 external_mini_cluster.cc:1428] Setting key 8589667031cba8ad27ac9ab257a9b263
I20260619 08:59:51.814112 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:51.805717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52517 (local address 127.0.98.62:35937)
0619 08:59:51.805776 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:51.805778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:51.805966 (+ 188us) server_negotiation.cc:407] Connection header received
0619 08:59:51.806874 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:51.806876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:51.806887 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:51.806948 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:51.807525 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:51.808012 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:51.809055 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:51.809251 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:51.809309 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:51.811349 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:51.811350 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:51.811350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:59:51.811357 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:51.812912 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:51.813271 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:51.813272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:51.813272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:51.813334 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:51.813533 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:51.813534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:51.813534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:51.813655 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:51.813708 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:51.813937 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 08:59:51.814020 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 08:59:51.814332 17759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:51.805681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35937 (local address 127.0.98.2:52517)
0619 08:59:51.805851 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:51.805891 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:51.806755 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:51.807003 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:51.807006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:51.807259 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:59:51.807452 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:51.807457 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:51.808098 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:51.808098 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:51.808968 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:51.808971 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:51.809322 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:51.809323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:51.809423 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:51.809911 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:59:51.809919 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:59:51.811255 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:59:51.812983 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:59:51.812989 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:59:51.812990 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:59:51.813200 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:59:51.813390 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:59:51.813390 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:59:51.813390 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:59:51.813487 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:59:51.813706 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:59:51.813709 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:59:51.813824 (+ 115us) client_negotiation.cc:769] Sending connection context
0619 08:59:51.813914 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 08:59:51.814091 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
I20260619 08:59:51.814740 17757 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35937
I20260619 08:59:51.814842 17757 heartbeater.cc:464] Registering TS with master...
I20260619 08:59:51.815016 17757 heartbeater.cc:511] Master 127.0.98.62:35937 requested a full tablet report, sending...
I20260619 08:59:51.815395 17432 ts_manager.cc:194] Registered new tserver with Master: 8ea59a825de24fe596e72b871f7f23f2 (127.0.98.2:35837)
I20260619 08:59:51.815778 17432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52517
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 08:59:51.834362 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35937
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:59:51.916263 17762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:59:51.916472 17762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:59:51.916494 17762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:59:51.917599 17762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:59:51.917632 17762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:59:51.917657 17762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 08:59:51.919101 17762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:59:51.919291 17762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:59:51.919476 17762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:59:51.924124 17767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:51.924142 17768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:51.924266 17770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:51.924158 17762 server_base.cc:1034] running on GCE node
I20260619 08:59:51.924530 17762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:59:51.924774 17762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:59:51.925940 17762 hybrid_clock.cc:648] HybridClock initialized: now 1781859591925921 us; error 34 us; skew 500 ppm
Jun 19 08:59:51 dist-test-slave-h4g5 krb5kdc[17387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:59:51.928318 17762 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 08:59:51.928750 17762 webserver.cc:460] Webserver started at http://127.0.98.3:37169/ using document root <none> and password file <none>
I20260619 08:59:51.928965 17762 fs_manager.cc:359] Metadata directory not provided
I20260619 08:59:51.929008 17762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:59:51.929133 17762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:59:51.930142 17762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-2/data/instance:
uuid: "516056e017644f07a780409adbed3198"
format_stamp: "Formatted at 2026-06-19 08:59:51 on dist-test-slave-h4g5"
server_key: "cebb9fea91a5becb355cc54695db0149"
server_key_iv: "b371db3f0114744c1b3bbba2a812860a"
server_key_version: "encryptionkey@0"
I20260619 08:59:51.930452 17762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-2/wal/instance:
uuid: "516056e017644f07a780409adbed3198"
format_stamp: "Formatted at 2026-06-19 08:59:51 on dist-test-slave-h4g5"
server_key: "cebb9fea91a5becb355cc54695db0149"
server_key_iv: "b371db3f0114744c1b3bbba2a812860a"
server_key_version: "encryptionkey@0"
I20260619 08:59:51.931632 17762 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:59:51.932302 17777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:51.932438 17762 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:59:51.932514 17762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-2/wal
uuid: "516056e017644f07a780409adbed3198"
format_stamp: "Formatted at 2026-06-19 08:59:51 on dist-test-slave-h4g5"
server_key: "cebb9fea91a5becb355cc54695db0149"
server_key_iv: "b371db3f0114744c1b3bbba2a812860a"
server_key_version: "encryptionkey@0"
I20260619 08:59:51.932576 17762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:59:51.948987 17762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:59:51.951087 17762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:59:51.951206 17762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:59:51.951444 17762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:59:51.951745 17762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:59:51.951776 17762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:51.951798 17762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:59:51.951809 17762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:51.957087 17762 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41337
I20260619 08:59:51.957137 17890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41337 every 8 connection(s)
I20260619 08:59:51.957432 17762 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
I20260619 08:59:51.959614 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 17762
I20260619 08:59:51.959705 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-2/wal/instance
I20260619 08:59:51.959868 392 external_mini_cluster.cc:1428] Setting key e491b5c0bb8f94e11f76ef6cbff12b63
Jun 19 08:59:51 dist-test-slave-h4g5 krb5kdc[17387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:59:51.966140 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:51.958070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39115 (local address 127.0.98.62:35937)
0619 08:59:51.958128 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:51.958130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:51.958306 (+ 176us) server_negotiation.cc:407] Connection header received
0619 08:59:51.959253 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:51.959255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:51.959266 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:51.959331 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:51.959982 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:51.960625 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:51.961608 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:51.961814 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:51.961870 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:51.963780 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:51.963782 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:51.963782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:59:51.963789 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:51.965085 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:51.965455 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:51.965455 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:51.965456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:51.965503 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:51.965681 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:51.965681 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:51.965681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:51.965772 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:51.965819 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:51.966021 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 08:59:51.966075 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 08:59:51.966337 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:51.958057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35937 (local address 127.0.98.3:39115)
0619 08:59:51.958199 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:51.958234 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:51.959160 (+ 926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:51.959390 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:51.959393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:51.959648 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:59:51.959876 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:51.959881 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:51.960736 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:51.960737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:51.961535 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:51.961537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:51.961876 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:51.961877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:51.961971 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:51.962392 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:59:51.962399 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:59:51.963705 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:59:51.965159 (+ 1454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:59:51.965165 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:59:51.965166 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:59:51.965388 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:59:51.965548 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:59:51.965548 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:59:51.965549 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:59:51.965640 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:59:51.965814 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:59:51.965817 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:59:51.965931 (+ 114us) client_negotiation.cc:769] Sending connection context
0619 08:59:51.966009 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 08:59:51.966126 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260619 08:59:51.966737 17891 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35937
I20260619 08:59:51.966830 17891 heartbeater.cc:464] Registering TS with master...
I20260619 08:59:51.966995 17891 heartbeater.cc:511] Master 127.0.98.62:35937 requested a full tablet report, sending...
I20260619 08:59:51.967350 17432 ts_manager.cc:194] Registered new tserver with Master: 516056e017644f07a780409adbed3198 (127.0.98.3:41337)
I20260619 08:59:51.967686 17432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39115
I20260619 08:59:51.971731 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:59:51.978613 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:51.973108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54352 (local address 127.0.98.62:35937)
0619 08:59:51.973147 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:51.973149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:51.973269 (+ 120us) server_negotiation.cc:407] Connection header received
0619 08:59:51.973330 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:51.973331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:51.973341 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:51.973383 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:51.973934 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:51.974401 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:51.975095 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:51.975287 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:51.975340 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:51.976467 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:51.976468 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:51.976469 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:59:51.976475 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:51.977596 (+ 1121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:51.977891 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:51.977892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:51.977893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:51.977944 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:51.978106 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:51.978107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:51.978107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:51.978264 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:51.978355 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:51.978450 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 08:59:51.978525 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 08:59:51.980008 17432 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:59:51.981580 17432 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:59:51.987867 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:51.983639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35837 (local address 127.0.0.1:38524)
0619 08:59:51.983761 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:51.983769 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:51.983804 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:51.984235 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:51.984235 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:51.984237 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:59:51.984274 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:51.984275 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:51.985633 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:51.985634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:51.986651 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:51.986654 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:51.987579 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:51.987580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:51.987649 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:51.987662 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 08:59:51.987728 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 08:59:51.987779 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":71,"thread_start_us":48,"threads_started":1}
I20260619 08:59:51.987985 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:51.983761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35319 (local address 127.0.0.1:51816)
0619 08:59:51.984352 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:51.984362 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:51.984412 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:51.984667 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:51.984668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:51.984669 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:59:51.984708 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:51.984709 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:51.985537 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:51.985537 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:51.986881 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:51.986884 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:51.987695 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:51.987696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:51.987747 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:51.987751 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:59:51.987794 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 08:59:51.987844 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":43,"thread_start_us":85,"threads_started":1}
I20260619 08:59:51.988075 17904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:51.983878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51816 (local address 127.0.98.1:35319)
0619 08:59:51.984031 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:51.984034 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:51.984465 (+ 431us) server_negotiation.cc:407] Connection header received
0619 08:59:51.984509 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:51.984512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:51.984527 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:51.984592 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:59:51.984771 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:51.985450 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:51.986951 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:51.987627 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:51.987688 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:51.987738 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:51.987912 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 08:59:51.987957 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260619 08:59:51.987910 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:51.983760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38524 (local address 127.0.98.2:35837)
0619 08:59:51.984062 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:51.984065 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:51.984074 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:59:51.984134 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:51.984136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:51.984150 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:51.984218 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:59:51.984419 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:51.985517 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:51.986745 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:51.987503 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:51.987571 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:51.987631 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:51.987757 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 08:59:51.987832 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":92,"threads_started":1}
I20260619 08:59:51.988830 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:51.983450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41337 (local address 127.0.0.1:49044)
0619 08:59:51.983564 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:51.983603 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:51.983666 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:51.984178 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:51.984182 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:51.984184 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:59:51.984256 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:51.984258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:51.985609 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:51.985610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:51.986881 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:51.986884 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:51.988623 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:51.988624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:51.988682 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:51.988686 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:59:51.988714 (+ 28us) client_negotiation.cc:240] Negotiation successful
0619 08:59:51.988750 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":48,"thread_start_us":52,"threads_started":1}
I20260619 08:59:51.988787 17557 tablet_service.cc:1467] Processing CreateTablet for tablet b8d78f2d8c4242758e7efd9d49bd19f3 (DEFAULT_TABLE table=test-table [id=1bd23cee6ac94bafb802809a9da237b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:59:51.989108 17557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8d78f2d8c4242758e7efd9d49bd19f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:59:51.989005 17691 tablet_service.cc:1467] Processing CreateTablet for tablet b8d78f2d8c4242758e7efd9d49bd19f3 (DEFAULT_TABLE table=test-table [id=1bd23cee6ac94bafb802809a9da237b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:59:51.989188 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:51.983546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49044 (local address 127.0.98.3:41337)
0619 08:59:51.983979 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:51.983984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:51.983995 (+ 11us) server_negotiation.cc:407] Connection header received
0619 08:59:51.984035 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:51.984037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:51.984054 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:51.984182 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:59:51.984397 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:51.985520 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:51.987775 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:51.988530 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:51.988988 (+ 458us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:51.989054 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:51.989078 (+ 24us) server_negotiation.cc:299] Negotiation successful
0619 08:59:51.989115 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":100,"threads_started":1}
I20260619 08:59:51.989324 17691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8d78f2d8c4242758e7efd9d49bd19f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:59:51.989820 17825 tablet_service.cc:1467] Processing CreateTablet for tablet b8d78f2d8c4242758e7efd9d49bd19f3 (DEFAULT_TABLE table=test-table [id=1bd23cee6ac94bafb802809a9da237b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:59:51.990123 17825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8d78f2d8c4242758e7efd9d49bd19f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:59:51.992158 17905 tablet_bootstrap.cc:492] T b8d78f2d8c4242758e7efd9d49bd19f3 P 8ea59a825de24fe596e72b871f7f23f2: Bootstrap starting.
I20260619 08:59:51.992789 17905 tablet_bootstrap.cc:654] T b8d78f2d8c4242758e7efd9d49bd19f3 P 8ea59a825de24fe596e72b871f7f23f2: Neither blocks nor log segments found. Creating new log.
I20260619 08:59:51.992974 17907 tablet_bootstrap.cc:492] T b8d78f2d8c4242758e7efd9d49bd19f3 P 516056e017644f07a780409adbed3198: Bootstrap starting.
I20260619 08:59:51.993304 17905 log.cc:824] T b8d78f2d8c4242758e7efd9d49bd19f3 P 8ea59a825de24fe596e72b871f7f23f2: Log is configured to *not* fsync() on all Append() calls
I20260619 08:59:51.993546 17907 tablet_bootstrap.cc:654] T b8d78f2d8c4242758e7efd9d49bd19f3 P 516056e017644f07a780409adbed3198: Neither blocks nor log segments found. Creating new log.
I20260619 08:59:51.993613 17906 tablet_bootstrap.cc:492] T b8d78f2d8c4242758e7efd9d49bd19f3 P 53063539bae848788059f09418c88336: Bootstrap starting.
I20260619 08:59:51.993938 17905 tablet_bootstrap.cc:492] T b8d78f2d8c4242758e7efd9d49bd19f3 P 8ea59a825de24fe596e72b871f7f23f2: No bootstrap required, opened a new log
I20260619 08:59:51.993988 17905 ts_tablet_manager.cc:1397] T b8d78f2d8c4242758e7efd9d49bd19f3 P 8ea59a825de24fe596e72b871f7f23f2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:59:51.994024 17907 log.cc:824] T b8d78f2d8c4242758e7efd9d49bd19f3 P 516056e017644f07a780409adbed3198: Log is configured to *not* fsync() on all Append() calls
I20260619 08:59:51.994616 17907 tablet_bootstrap.cc:492] T b8d78f2d8c4242758e7efd9d49bd19f3 P 516056e017644f07a780409adbed3198: No bootstrap required, opened a new log
I20260619 08:59:51.994668 17907 ts_tablet_manager.cc:1397] T b8d78f2d8c4242758e7efd9d49bd19f3 P 516056e017644f07a780409adbed3198: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:59:51.995513 17905 raft_consensus.cc:348] T b8d78f2d8c4242758e7efd9d49bd19f3 P 8ea59a825de24fe596e72b871f7f23f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516056e017644f07a780409adbed3198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41337 } } peers { permanent_uuid: "8ea59a825de24fe596e72b871f7f23f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35837 } } peers { permanent_uuid: "53063539bae848788059f09418c88336" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35319 } }
I20260619 08:59:51.995647 17905 raft_consensus.cc:374] T b8d78f2d8c4242758e7efd9d49bd19f3 P 8ea59a825de24fe596e72b871f7f23f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:59:51.995678 17905 raft_consensus.cc:729] T b8d78f2d8c4242758e7efd9d49bd19f3 P 8ea59a825de24fe596e72b871f7f23f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ea59a825de24fe596e72b871f7f23f2, State: Initialized, Role: FOLLOWER
I20260619 08:59:51.995790 17905 consensus_queue.cc:260] T b8d78f2d8c4242758e7efd9d49bd19f3 P 8ea59a825de24fe596e72b871f7f23f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516056e017644f07a780409adbed3198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41337 } } peers { permanent_uuid: "8ea59a825de24fe596e72b871f7f23f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35837 } } peers { permanent_uuid: "53063539bae848788059f09418c88336" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35319 } }
I20260619 08:59:51.996027 17757 heartbeater.cc:503] Master 127.0.98.62:35937 was elected leader, sending a full tablet report...
I20260619 08:59:51.996282 17905 ts_tablet_manager.cc:1428] T b8d78f2d8c4242758e7efd9d49bd19f3 P 8ea59a825de24fe596e72b871f7f23f2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:59:51.996210 17907 raft_consensus.cc:348] T b8d78f2d8c4242758e7efd9d49bd19f3 P 516056e017644f07a780409adbed3198 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516056e017644f07a780409adbed3198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41337 } } peers { permanent_uuid: "8ea59a825de24fe596e72b871f7f23f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35837 } } peers { permanent_uuid: "53063539bae848788059f09418c88336" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35319 } }
I20260619 08:59:51.996337 17907 raft_consensus.cc:374] T b8d78f2d8c4242758e7efd9d49bd19f3 P 516056e017644f07a780409adbed3198 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:59:51.996415 17907 raft_consensus.cc:729] T b8d78f2d8c4242758e7efd9d49bd19f3 P 516056e017644f07a780409adbed3198 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516056e017644f07a780409adbed3198, State: Initialized, Role: FOLLOWER
I20260619 08:59:51.996534 17907 consensus_queue.cc:260] T b8d78f2d8c4242758e7efd9d49bd19f3 P 516056e017644f07a780409adbed3198 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516056e017644f07a780409adbed3198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41337 } } peers { permanent_uuid: "8ea59a825de24fe596e72b871f7f23f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35837 } } peers { permanent_uuid: "53063539bae848788059f09418c88336" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35319 } }
I20260619 08:59:51.996791 17891 heartbeater.cc:503] Master 127.0.98.62:35937 was elected leader, sending a full tablet report...
I20260619 08:59:51.996932 17906 tablet_bootstrap.cc:654] T b8d78f2d8c4242758e7efd9d49bd19f3 P 53063539bae848788059f09418c88336: Neither blocks nor log segments found. Creating new log.
I20260619 08:59:51.996793 17907 ts_tablet_manager.cc:1428] T b8d78f2d8c4242758e7efd9d49bd19f3 P 516056e017644f07a780409adbed3198: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 08:59:51.997444 17906 log.cc:824] T b8d78f2d8c4242758e7efd9d49bd19f3 P 53063539bae848788059f09418c88336: Log is configured to *not* fsync() on all Append() calls
I20260619 08:59:51.998098 17906 tablet_bootstrap.cc:492] T b8d78f2d8c4242758e7efd9d49bd19f3 P 53063539bae848788059f09418c88336: No bootstrap required, opened a new log
I20260619 08:59:51.998152 17906 ts_tablet_manager.cc:1397] T b8d78f2d8c4242758e7efd9d49bd19f3 P 53063539bae848788059f09418c88336: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260619 08:59:51.999614 17906 raft_consensus.cc:348] T b8d78f2d8c4242758e7efd9d49bd19f3 P 53063539bae848788059f09418c88336 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516056e017644f07a780409adbed3198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41337 } } peers { permanent_uuid: "8ea59a825de24fe596e72b871f7f23f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35837 } } peers { permanent_uuid: "53063539bae848788059f09418c88336" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35319 } }
I20260619 08:59:51.999750 17906 raft_consensus.cc:374] T b8d78f2d8c4242758e7efd9d49bd19f3 P 53063539bae848788059f09418c88336 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:59:51.999771 17906 raft_consensus.cc:729] T b8d78f2d8c4242758e7efd9d49bd19f3 P 53063539bae848788059f09418c88336 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53063539bae848788059f09418c88336, State: Initialized, Role: FOLLOWER
I20260619 08:59:51.999866 17906 consensus_queue.cc:260] T b8d78f2d8c4242758e7efd9d49bd19f3 P 53063539bae848788059f09418c88336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516056e017644f07a780409adbed3198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41337 } } peers { permanent_uuid: "8ea59a825de24fe596e72b871f7f23f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35837 } } peers { permanent_uuid: "53063539bae848788059f09418c88336" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35319 } }
I20260619 08:59:52.000164 17906 ts_tablet_manager.cc:1428] T b8d78f2d8c4242758e7efd9d49bd19f3 P 53063539bae848788059f09418c88336: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:59:52.000159 17623 heartbeater.cc:503] Master 127.0.98.62:35937 was elected leader, sending a full tablet report...
W20260619 08:59:52.055606 17758 tablet.cc:2369] T b8d78f2d8c4242758e7efd9d49bd19f3 P 8ea59a825de24fe596e72b871f7f23f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:59:52.080492 17910 raft_consensus.cc:482] T b8d78f2d8c4242758e7efd9d49bd19f3 P 8ea59a825de24fe596e72b871f7f23f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:59:52.080610 17910 raft_consensus.cc:504] T b8d78f2d8c4242758e7efd9d49bd19f3 P 8ea59a825de24fe596e72b871f7f23f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516056e017644f07a780409adbed3198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41337 } } peers { permanent_uuid: "8ea59a825de24fe596e72b871f7f23f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35837 } } peers { permanent_uuid: "53063539bae848788059f09418c88336" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35319 } }
I20260619 08:59:52.080971 17910 leader_election.cc:288] T b8d78f2d8c4242758e7efd9d49bd19f3 P 8ea59a825de24fe596e72b871f7f23f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516056e017644f07a780409adbed3198 (127.0.98.3:41337), 53063539bae848788059f09418c88336 (127.0.98.1:35319)
I20260619 08:59:52.083673 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:52.081199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34319 (local address 127.0.98.3:41337)
0619 08:59:52.081286 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:52.081288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:52.081297 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:59:52.081321 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:52.081322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:52.081330 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:52.081380 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:59:52.081545 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.082058 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.083008 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.083369 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.083427 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:52.083438 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:52.083573 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 08:59:52.083608 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 08:59:52.083681 17904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:52.081301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39551 (local address 127.0.98.1:35319)
0619 08:59:52.081343 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:52.081345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:52.081438 (+ 93us) server_negotiation.cc:407] Connection header received
0619 08:59:52.081469 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:52.081469 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:52.081481 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:52.081537 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:59:52.081669 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.082101 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.083003 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.083372 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.083430 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:52.083440 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:52.083584 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 08:59:52.083621 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 08:59:52.083680 17759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:52.081125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41337 (local address 127.0.98.2:34319)
0619 08:59:52.081179 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:52.081192 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:52.081250 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:52.081397 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:52.081398 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:52.081400 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:59:52.081474 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:52.081476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.082133 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.082134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:52.082922 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:52.082925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.083442 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.083444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:52.083511 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:52.083514 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:59:52.083556 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 08:59:52.083597 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":17}
I20260619 08:59:52.083690 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:52.081218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35319 (local address 127.0.98.2:39551)
0619 08:59:52.081373 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:52.081381 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:52.081428 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:52.081544 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:52.081545 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:52.081546 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:59:52.081613 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:52.081615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.082174 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.082175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:52.082921 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:52.082923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.083447 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.083448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:52.083519 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:52.083521 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:59:52.083570 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 08:59:52.083608 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":142,"threads_started":1}
I20260619 08:59:52.084056 17845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8d78f2d8c4242758e7efd9d49bd19f3" candidate_uuid: "8ea59a825de24fe596e72b871f7f23f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516056e017644f07a780409adbed3198" is_pre_election: true
I20260619 08:59:52.084250 17845 raft_consensus.cc:2454] T b8d78f2d8c4242758e7efd9d49bd19f3 P 516056e017644f07a780409adbed3198 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ea59a825de24fe596e72b871f7f23f2 in term 0.
I20260619 08:59:52.084286 17577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8d78f2d8c4242758e7efd9d49bd19f3" candidate_uuid: "8ea59a825de24fe596e72b871f7f23f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53063539bae848788059f09418c88336" is_pre_election: true
I20260619 08:59:52.084429 17577 raft_consensus.cc:2454] T b8d78f2d8c4242758e7efd9d49bd19f3 P 53063539bae848788059f09418c88336 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ea59a825de24fe596e72b871f7f23f2 in term 0.
I20260619 08:59:52.084460 17647 leader_election.cc:302] T b8d78f2d8c4242758e7efd9d49bd19f3 P 8ea59a825de24fe596e72b871f7f23f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516056e017644f07a780409adbed3198, 8ea59a825de24fe596e72b871f7f23f2; no voters:
I20260619 08:59:52.084640 17910 raft_consensus.cc:2781] T b8d78f2d8c4242758e7efd9d49bd19f3 P 8ea59a825de24fe596e72b871f7f23f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:59:52.084679 17910 raft_consensus.cc:482] T b8d78f2d8c4242758e7efd9d49bd19f3 P 8ea59a825de24fe596e72b871f7f23f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:59:52.084693 17910 raft_consensus.cc:3037] T b8d78f2d8c4242758e7efd9d49bd19f3 P 8ea59a825de24fe596e72b871f7f23f2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:59:52.085345 17910 raft_consensus.cc:504] T b8d78f2d8c4242758e7efd9d49bd19f3 P 8ea59a825de24fe596e72b871f7f23f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516056e017644f07a780409adbed3198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41337 } } peers { permanent_uuid: "8ea59a825de24fe596e72b871f7f23f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35837 } } peers { permanent_uuid: "53063539bae848788059f09418c88336" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35319 } }
I20260619 08:59:52.085454 17910 leader_election.cc:288] T b8d78f2d8c4242758e7efd9d49bd19f3 P 8ea59a825de24fe596e72b871f7f23f2 [CANDIDATE]: Term 1 election: Requested vote from peers 516056e017644f07a780409adbed3198 (127.0.98.3:41337), 53063539bae848788059f09418c88336 (127.0.98.1:35319)
I20260619 08:59:52.085613 17577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8d78f2d8c4242758e7efd9d49bd19f3" candidate_uuid: "8ea59a825de24fe596e72b871f7f23f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53063539bae848788059f09418c88336"
I20260619 08:59:52.085634 17845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8d78f2d8c4242758e7efd9d49bd19f3" candidate_uuid: "8ea59a825de24fe596e72b871f7f23f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516056e017644f07a780409adbed3198"
I20260619 08:59:52.085690 17845 raft_consensus.cc:3037] T b8d78f2d8c4242758e7efd9d49bd19f3 P 516056e017644f07a780409adbed3198 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:59:52.085687 17577 raft_consensus.cc:3037] T b8d78f2d8c4242758e7efd9d49bd19f3 P 53063539bae848788059f09418c88336 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:59:52.086359 17845 raft_consensus.cc:2454] T b8d78f2d8c4242758e7efd9d49bd19f3 P 516056e017644f07a780409adbed3198 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ea59a825de24fe596e72b871f7f23f2 in term 1.
I20260619 08:59:52.086371 17577 raft_consensus.cc:2454] T b8d78f2d8c4242758e7efd9d49bd19f3 P 53063539bae848788059f09418c88336 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ea59a825de24fe596e72b871f7f23f2 in term 1.
I20260619 08:59:52.086504 17647 leader_election.cc:302] T b8d78f2d8c4242758e7efd9d49bd19f3 P 8ea59a825de24fe596e72b871f7f23f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516056e017644f07a780409adbed3198, 8ea59a825de24fe596e72b871f7f23f2; no voters:
I20260619 08:59:52.086596 17910 raft_consensus.cc:2781] T b8d78f2d8c4242758e7efd9d49bd19f3 P 8ea59a825de24fe596e72b871f7f23f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:59:52.086756 17910 raft_consensus.cc:686] T b8d78f2d8c4242758e7efd9d49bd19f3 P 8ea59a825de24fe596e72b871f7f23f2 [term 1 LEADER]: Becoming Leader. State: Replica: 8ea59a825de24fe596e72b871f7f23f2, State: Running, Role: LEADER
I20260619 08:59:52.086839 17910 consensus_queue.cc:237] T b8d78f2d8c4242758e7efd9d49bd19f3 P 8ea59a825de24fe596e72b871f7f23f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516056e017644f07a780409adbed3198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41337 } } peers { permanent_uuid: "8ea59a825de24fe596e72b871f7f23f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35837 } } peers { permanent_uuid: "53063539bae848788059f09418c88336" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35319 } }
I20260619 08:59:52.087643 17432 catalog_manager.cc:5495] T b8d78f2d8c4242758e7efd9d49bd19f3 P 8ea59a825de24fe596e72b871f7f23f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ea59a825de24fe596e72b871f7f23f2 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8ea59a825de24fe596e72b871f7f23f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516056e017644f07a780409adbed3198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ea59a825de24fe596e72b871f7f23f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53063539bae848788059f09418c88336" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35319 } health_report { overall_health: UNKNOWN } } }
I20260619 08:59:52.105786 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:52.103552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38532 (local address 127.0.98.2:35837)
0619 08:59:52.103609 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:52.103610 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:52.103617 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:59:52.103652 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:52.103653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:52.103673 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:52.103725 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:59:52.103828 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.104300 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.105035 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.105211 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.105255 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:52.105452 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:59:52.105548 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:59:52.105607 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:52.105678 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 08:59:52.105694 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 08:59:52.107590 17577 raft_consensus.cc:1261] T b8d78f2d8c4242758e7efd9d49bd19f3 P 53063539bae848788059f09418c88336 [term 1 FOLLOWER]: Refusing update from remote peer 8ea59a825de24fe596e72b871f7f23f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:59:52.107591 17845 raft_consensus.cc:1261] T b8d78f2d8c4242758e7efd9d49bd19f3 P 516056e017644f07a780409adbed3198 [term 1 FOLLOWER]: Refusing update from remote peer 8ea59a825de24fe596e72b871f7f23f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:59:52.107926 17910 consensus_queue.cc:1035] T b8d78f2d8c4242758e7efd9d49bd19f3 P 8ea59a825de24fe596e72b871f7f23f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53063539bae848788059f09418c88336" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:59:52.108095 17915 consensus_queue.cc:1035] T b8d78f2d8c4242758e7efd9d49bd19f3 P 8ea59a825de24fe596e72b871f7f23f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516056e017644f07a780409adbed3198" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:59:52.111160 17919 mvcc.cc:205] Tried to move back new op lower bound from 7298496889269653504 to 7298496889188118528. Current Snapshot: MvccSnapshot[applied={T|T < 7298496889269653504 or (T in {7298496889269653504})}]
I20260619 08:59:52.111301 17920 mvcc.cc:205] Tried to move back new op lower bound from 7298496889269653504 to 7298496889188118528. Current Snapshot: MvccSnapshot[applied={T|T < 7298496889269653504}]
I20260619 08:59:52.112540 17431 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:59:52.112632 17431 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:59:52.113389 17431 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 711bdd72c206433996e5fb9a844fde7f: Sending DeleteTablet for 3 replicas of tablet b8d78f2d8c4242758e7efd9d49bd19f3
I20260619 08:59:52.113678 17825 tablet_service.cc:1514] Processing DeleteTablet for tablet b8d78f2d8c4242758e7efd9d49bd19f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:59:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49044
I20260619 08:59:52.113745 17557 tablet_service.cc:1514] Processing DeleteTablet for tablet b8d78f2d8c4242758e7efd9d49bd19f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:59:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51816
I20260619 08:59:52.113808 17691 tablet_service.cc:1514] Processing DeleteTablet for tablet b8d78f2d8c4242758e7efd9d49bd19f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:59:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38524
I20260619 08:59:52.113993 17928 tablet_replica.cc:330] T b8d78f2d8c4242758e7efd9d49bd19f3 P 53063539bae848788059f09418c88336: stopping tablet replica
I20260619 08:59:52.114082 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 17494
I20260619 08:59:52.114673 17929 tablet_replica.cc:330] T b8d78f2d8c4242758e7efd9d49bd19f3 P 8ea59a825de24fe596e72b871f7f23f2: stopping tablet replica
I20260619 08:59:52.114773 17929 raft_consensus.cc:2229] T b8d78f2d8c4242758e7efd9d49bd19f3 P 8ea59a825de24fe596e72b871f7f23f2 [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:59:52.114854 17929 raft_consensus.cc:2258] T b8d78f2d8c4242758e7efd9d49bd19f3 P 8ea59a825de24fe596e72b871f7f23f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:59:52.115520 17929 ts_tablet_manager.cc:1905] T b8d78f2d8c4242758e7efd9d49bd19f3 P 8ea59a825de24fe596e72b871f7f23f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:59:52.116202 17927 tablet_replica.cc:330] T b8d78f2d8c4242758e7efd9d49bd19f3 P 516056e017644f07a780409adbed3198: stopping tablet replica
I20260619 08:59:52.116379 17927 raft_consensus.cc:2229] T b8d78f2d8c4242758e7efd9d49bd19f3 P 516056e017644f07a780409adbed3198 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:59:52.116492 17927 raft_consensus.cc:2258] T b8d78f2d8c4242758e7efd9d49bd19f3 P 516056e017644f07a780409adbed3198 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:59:52.116923 17927 ts_tablet_manager.cc:1905] T b8d78f2d8c4242758e7efd9d49bd19f3 P 516056e017644f07a780409adbed3198: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:59:52.116951 17929 ts_tablet_manager.cc:1918] T b8d78f2d8c4242758e7efd9d49bd19f3 P 8ea59a825de24fe596e72b871f7f23f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:59:52.117005 17929 log.cc:1196] T b8d78f2d8c4242758e7efd9d49bd19f3 P 8ea59a825de24fe596e72b871f7f23f2: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-1/wal/wals/b8d78f2d8c4242758e7efd9d49bd19f3
I20260619 08:59:52.117245 17929 ts_tablet_manager.cc:1939] T b8d78f2d8c4242758e7efd9d49bd19f3 P 8ea59a825de24fe596e72b871f7f23f2: Deleting consensus metadata
I20260619 08:59:52.118283 17927 ts_tablet_manager.cc:1918] T b8d78f2d8c4242758e7efd9d49bd19f3 P 516056e017644f07a780409adbed3198: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:59:52.118359 17927 log.cc:1196] T b8d78f2d8c4242758e7efd9d49bd19f3 P 516056e017644f07a780409adbed3198: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859361540516-392-0/minicluster-data/ts-2/wal/wals/b8d78f2d8c4242758e7efd9d49bd19f3
I20260619 08:59:52.118500 17419 catalog_manager.cc:4841] TS 8ea59a825de24fe596e72b871f7f23f2 (127.0.98.2:35837): tablet b8d78f2d8c4242758e7efd9d49bd19f3 (table test-table [id=1bd23cee6ac94bafb802809a9da237b4]) successfully deleted
I20260619 08:59:52.118593 17927 ts_tablet_manager.cc:1939] T b8d78f2d8c4242758e7efd9d49bd19f3 P 516056e017644f07a780409adbed3198: Deleting consensus metadata
I20260619 08:59:52.118952 17419 catalog_manager.cc:4841] TS 516056e017644f07a780409adbed3198 (127.0.98.3:41337): tablet b8d78f2d8c4242758e7efd9d49bd19f3 (table test-table [id=1bd23cee6ac94bafb802809a9da237b4]) successfully deleted
W20260619 08:59:52.120298 17419 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35319 (error 108)
I20260619 08:59:52.120774 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:52.120630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35319 (local address 127.0.0.1:51826)
0619 08:59:52.120693 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:52.120740 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 08:59:52.120811 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 17628
W20260619 08:59:52.120878 17419 catalog_manager.cc:4584] TS 53063539bae848788059f09418c88336 (127.0.98.1:35319): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b8d78f2d8c4242758e7efd9d49bd19f3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35319: connect: Connection refused (error 111)
I20260619 08:59:52.125496 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 17762
I20260619 08:59:52.129202 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 17400
2026-06-19T08:59:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1861 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T08:59:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:59:52Z Disabled control of system clock
I20260619 08:59:52.154394 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42629
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 08:59:52.231317 17934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:59:52.231537 17934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:59:52.231562 17934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:59:52.232699 17934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:59:52.232739 17934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:59:52.232755 17934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:59:52.232771 17934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:59:52.232785 17934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:59:52.234118 17934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42629
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:59:52.234366 17934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:59:52.234629 17934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:59:52.239039 17940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:52.239030 17939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:52.239030 17942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:52.239703 17934 server_base.cc:1034] running on GCE node
I20260619 08:59:52.239941 17934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:59:52.240227 17934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:59:52.241385 17934 hybrid_clock.cc:648] HybridClock initialized: now 1781859592241371 us; error 29 us; skew 500 ppm
I20260619 08:59:52.242555 17934 webserver.cc:460] Webserver started at http://127.0.98.62:43921/ using document root <none> and password file <none>
I20260619 08:59:52.242770 17934 fs_manager.cc:359] Metadata directory not provided
I20260619 08:59:52.242818 17934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:59:52.242945 17934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:59:52.243960 17934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "143e37781d3d4d3f8521c5d587cce216"
format_stamp: "Formatted at 2026-06-19 08:59:52 on dist-test-slave-h4g5"
server_key: "6ce7d8ba463bf47b181fc7614df56c98"
server_key_iv: "c8d130091c4f3860f551d488902559c4"
server_key_version: "encryptionkey@0"
I20260619 08:59:52.244320 17934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "143e37781d3d4d3f8521c5d587cce216"
format_stamp: "Formatted at 2026-06-19 08:59:52 on dist-test-slave-h4g5"
server_key: "6ce7d8ba463bf47b181fc7614df56c98"
server_key_iv: "c8d130091c4f3860f551d488902559c4"
server_key_version: "encryptionkey@0"
I20260619 08:59:52.245497 17934 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:59:52.246161 17948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:52.246297 17934 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:59:52.246376 17934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "143e37781d3d4d3f8521c5d587cce216"
format_stamp: "Formatted at 2026-06-19 08:59:52 on dist-test-slave-h4g5"
server_key: "6ce7d8ba463bf47b181fc7614df56c98"
server_key_iv: "c8d130091c4f3860f551d488902559c4"
server_key_version: "encryptionkey@0"
I20260619 08:59:52.246446 17934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:59:52.253450 17934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:59:52.253736 17934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:59:52.253860 17934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:59:52.257550 17934 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42629
I20260619 08:59:52.257603 18000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42629 every 8 connection(s)
I20260619 08:59:52.257899 17934 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:59:52.258219 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 17934
I20260619 08:59:52.258311 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 08:59:52.258502 392 external_mini_cluster.cc:1428] Setting key 46cdf2906c11de513235ed4b67df46b2
I20260619 08:59:52.258993 18001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:59:52.262178 18001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 143e37781d3d4d3f8521c5d587cce216: Bootstrap starting.
I20260619 08:59:52.262730 18001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 143e37781d3d4d3f8521c5d587cce216: Neither blocks nor log segments found. Creating new log.
I20260619 08:59:52.262996 18001 log.cc:824] T 00000000000000000000000000000000 P 143e37781d3d4d3f8521c5d587cce216: Log is configured to *not* fsync() on all Append() calls
I20260619 08:59:52.263607 18001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 143e37781d3d4d3f8521c5d587cce216: No bootstrap required, opened a new log
I20260619 08:59:52.264171 18004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:52.259050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36324 (local address 127.0.98.62:42629)
0619 08:59:52.259258 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:52.259266 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:52.259298 (+ 32us) server_negotiation.cc:407] Connection header received
0619 08:59:52.259793 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:52.259802 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:52.259883 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:52.260029 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:52.260229 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.261561 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.262283 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.262706 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.262819 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:52.263490 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:52.263494 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:52.263495 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:59:52.263652 (+ 157us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:52.263686 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:59:52.263702 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:52.263756 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:52.263867 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 08:59:52.263936 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":91,"threads_started":1}
I20260619 08:59:52.265668 18001 raft_consensus.cc:348] T 00000000000000000000000000000000 P 143e37781d3d4d3f8521c5d587cce216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "143e37781d3d4d3f8521c5d587cce216" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42629 } }
I20260619 08:59:52.265799 18001 raft_consensus.cc:374] T 00000000000000000000000000000000 P 143e37781d3d4d3f8521c5d587cce216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:59:52.265832 18001 raft_consensus.cc:729] T 00000000000000000000000000000000 P 143e37781d3d4d3f8521c5d587cce216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 143e37781d3d4d3f8521c5d587cce216, State: Initialized, Role: FOLLOWER
I20260619 08:59:52.265945 18001 consensus_queue.cc:260] T 00000000000000000000000000000000 P 143e37781d3d4d3f8521c5d587cce216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "143e37781d3d4d3f8521c5d587cce216" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42629 } }
I20260619 08:59:52.266016 18001 raft_consensus.cc:388] T 00000000000000000000000000000000 P 143e37781d3d4d3f8521c5d587cce216 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:59:52.266047 18001 raft_consensus.cc:482] T 00000000000000000000000000000000 P 143e37781d3d4d3f8521c5d587cce216 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:59:52.266091 18001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 143e37781d3d4d3f8521c5d587cce216 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:59:52.266600 18001 raft_consensus.cc:504] T 00000000000000000000000000000000 P 143e37781d3d4d3f8521c5d587cce216 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "143e37781d3d4d3f8521c5d587cce216" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42629 } }
I20260619 08:59:52.266690 18001 leader_election.cc:302] T 00000000000000000000000000000000 P 143e37781d3d4d3f8521c5d587cce216 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 143e37781d3d4d3f8521c5d587cce216; no voters:
I20260619 08:59:52.266847 18001 leader_election.cc:288] T 00000000000000000000000000000000 P 143e37781d3d4d3f8521c5d587cce216 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:59:52.266937 18006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 143e37781d3d4d3f8521c5d587cce216 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:59:52.267096 18001 sys_catalog.cc:564] T 00000000000000000000000000000000 P 143e37781d3d4d3f8521c5d587cce216 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:59:52.267127 18006 raft_consensus.cc:686] T 00000000000000000000000000000000 P 143e37781d3d4d3f8521c5d587cce216 [term 1 LEADER]: Becoming Leader. State: Replica: 143e37781d3d4d3f8521c5d587cce216, State: Running, Role: LEADER
I20260619 08:59:52.267207 18006 consensus_queue.cc:237] T 00000000000000000000000000000000 P 143e37781d3d4d3f8521c5d587cce216 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "143e37781d3d4d3f8521c5d587cce216" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42629 } }
I20260619 08:59:52.267563 18007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 143e37781d3d4d3f8521c5d587cce216 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "143e37781d3d4d3f8521c5d587cce216" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "143e37781d3d4d3f8521c5d587cce216" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42629 } } }
I20260619 08:59:52.267637 18007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 143e37781d3d4d3f8521c5d587cce216 [sys.catalog]: This master's current role is: LEADER
I20260619 08:59:52.267962 18015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:59:52.268438 18015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:59:52.268682 18008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 143e37781d3d4d3f8521c5d587cce216 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 143e37781d3d4d3f8521c5d587cce216. Latest consensus state: current_term: 1 leader_uuid: "143e37781d3d4d3f8521c5d587cce216" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "143e37781d3d4d3f8521c5d587cce216" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42629 } } }
I20260619 08:59:52.268831 18008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 143e37781d3d4d3f8521c5d587cce216 [sys.catalog]: This master's current role is: LEADER
I20260619 08:59:52.273092 18015 catalog_manager.cc:1347] Generated new cluster ID: d8a098808b964216a5e6950e5107c318
I20260619 08:59:52.273154 18015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:59:52.294808 18015 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:59:52.296168 18015 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:59:52.302429 18015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 143e37781d3d4d3f8521c5d587cce216: Generated new TSK 0
I20260619 08:59:52.302637 18015 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:59:52.312704 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42629
--builtin_ntp_servers=127.0.98.20:33705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 08:59:52.394063 18025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:59:52.394260 18025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:59:52.394289 18025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:59:52.395421 18025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:59:52.395459 18025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:59:52.395493 18025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 08:59:52.396878 18025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:59:52.397106 18025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:59:52.397403 18025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:59:52.403009 18030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:52.403100 18033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:52.403347 18025 server_base.cc:1034] running on GCE node
W20260619 08:59:52.403033 18031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:52.403578 18025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:59:52.403787 18025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:59:52.404925 18025 hybrid_clock.cc:648] HybridClock initialized: now 1781859592404917 us; error 33 us; skew 500 ppm
I20260619 08:59:52.406132 18025 webserver.cc:460] Webserver started at http://127.0.98.1:39133/ using document root <none> and password file <none>
I20260619 08:59:52.406379 18025 fs_manager.cc:359] Metadata directory not provided
I20260619 08:59:52.406435 18025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:59:52.406569 18025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:59:52.407667 18025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-0/data/instance:
uuid: "1c1df67162f243648ff81e2a4aec3911"
format_stamp: "Formatted at 2026-06-19 08:59:52 on dist-test-slave-h4g5"
server_key: "b28a63201b3545b56714f4877cfcf765"
server_key_iv: "8931b4ddfbf2cbe9ba636641e9cbacd7"
server_key_version: "encryptionkey@0"
I20260619 08:59:52.407999 18025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1c1df67162f243648ff81e2a4aec3911"
format_stamp: "Formatted at 2026-06-19 08:59:52 on dist-test-slave-h4g5"
server_key: "b28a63201b3545b56714f4877cfcf765"
server_key_iv: "8931b4ddfbf2cbe9ba636641e9cbacd7"
server_key_version: "encryptionkey@0"
I20260619 08:59:52.409197 18025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:59:52.409917 18039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:52.410049 18025 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 08:59:52.410128 18025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-0/wal
uuid: "1c1df67162f243648ff81e2a4aec3911"
format_stamp: "Formatted at 2026-06-19 08:59:52 on dist-test-slave-h4g5"
server_key: "b28a63201b3545b56714f4877cfcf765"
server_key_iv: "8931b4ddfbf2cbe9ba636641e9cbacd7"
server_key_version: "encryptionkey@0"
I20260619 08:59:52.410192 18025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:59:52.420217 18025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:59:52.420490 18025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:59:52.420617 18025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:59:52.420816 18025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:59:52.421120 18025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:59:52.421154 18025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:52.421181 18025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:59:52.421203 18025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:52.426532 18025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35059
I20260619 08:59:52.426569 18152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35059 every 8 connection(s)
I20260619 08:59:52.426867 18025 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
I20260619 08:59:52.427716 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 18025
I20260619 08:59:52.427793 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-0/wal/instance
I20260619 08:59:52.427989 392 external_mini_cluster.cc:1428] Setting key 98a0490a311f6f9f4d3edead56d6dd4f
I20260619 08:59:52.428892 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42629
--builtin_ntp_servers=127.0.98.20:33705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 08:59:52.432749 18004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:52.427533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47703 (local address 127.0.98.62:42629)
0619 08:59:52.427603 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:52.427606 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:52.428011 (+ 405us) server_negotiation.cc:407] Connection header received
0619 08:59:52.428562 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:52.428562 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:52.428577 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:52.428626 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:52.429354 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.430081 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.431332 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.431621 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.431683 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:52.432473 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:52.432475 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:52.432475 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:59:52.432481 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:52.432498 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:59:52.432502 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:52.432568 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:52.432626 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 08:59:52.432663 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 08:59:52.432879 18155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:52.427503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42629 (local address 127.0.98.1:47703)
0619 08:59:52.427889 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:52.427930 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:52.428490 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:52.428708 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:52.428712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:52.428721 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:59:52.429271 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:52.429276 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.430176 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.430177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:52.431254 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:52.431257 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.431693 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.431694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:52.431798 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:52.432345 (+ 547us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:59:52.432345 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:59:52.432374 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:59:52.432397 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:59:52.432569 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:59:52.432571 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:59:52.432573 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:59:52.432617 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 08:59:52.432692 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":77,"threads_started":1}
I20260619 08:59:52.433395 18153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42629
I20260619 08:59:52.433482 18153 heartbeater.cc:464] Registering TS with master...
I20260619 08:59:52.433651 18153 heartbeater.cc:511] Master 127.0.98.62:42629 requested a full tablet report, sending...
I20260619 08:59:52.434122 17965 ts_manager.cc:194] Registered new tserver with Master: 1c1df67162f243648ff81e2a4aec3911 (127.0.98.1:35059)
I20260619 08:59:52.434703 17965 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:47703
W20260619 08:59:52.510651 18156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:59:52.510850 18156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:59:52.510869 18156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:59:52.511984 18156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:59:52.512022 18156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:59:52.512045 18156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 08:59:52.513481 18156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:59:52.513703 18156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:59:52.513968 18156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:59:52.518677 18161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:52.518755 18164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:52.518930 18156 server_base.cc:1034] running on GCE node
W20260619 08:59:52.519029 18162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:52.519237 18156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:59:52.519503 18156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:59:52.520669 18156 hybrid_clock.cc:648] HybridClock initialized: now 1781859592520657 us; error 31 us; skew 500 ppm
I20260619 08:59:52.521981 18156 webserver.cc:460] Webserver started at http://127.0.98.2:43911/ using document root <none> and password file <none>
I20260619 08:59:52.522207 18156 fs_manager.cc:359] Metadata directory not provided
I20260619 08:59:52.522292 18156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:59:52.522426 18156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:59:52.523530 18156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-1/data/instance:
uuid: "d6d046ed529a4b09b0923f9fd46095cc"
format_stamp: "Formatted at 2026-06-19 08:59:52 on dist-test-slave-h4g5"
server_key: "0205bb0636619d304101e9c0e3d300ff"
server_key_iv: "7b12cb21b816873d6f3a1925adec07f7"
server_key_version: "encryptionkey@0"
I20260619 08:59:52.523877 18156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d6d046ed529a4b09b0923f9fd46095cc"
format_stamp: "Formatted at 2026-06-19 08:59:52 on dist-test-slave-h4g5"
server_key: "0205bb0636619d304101e9c0e3d300ff"
server_key_iv: "7b12cb21b816873d6f3a1925adec07f7"
server_key_version: "encryptionkey@0"
I20260619 08:59:52.525137 18156 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:59:52.525830 18170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:52.525969 18156 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 08:59:52.526053 18156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-1/wal
uuid: "d6d046ed529a4b09b0923f9fd46095cc"
format_stamp: "Formatted at 2026-06-19 08:59:52 on dist-test-slave-h4g5"
server_key: "0205bb0636619d304101e9c0e3d300ff"
server_key_iv: "7b12cb21b816873d6f3a1925adec07f7"
server_key_version: "encryptionkey@0"
I20260619 08:59:52.526116 18156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:59:52.548020 18156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:59:52.548429 18156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:59:52.548583 18156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:59:52.548821 18156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:59:52.549155 18156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:59:52.549198 18156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:52.549249 18156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:59:52.549289 18156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:52.555223 18156 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36393
I20260619 08:59:52.555284 18283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36393 every 8 connection(s)
I20260619 08:59:52.555624 18156 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
I20260619 08:59:52.561324 18004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:52.556351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46275 (local address 127.0.98.62:42629)
0619 08:59:52.556418 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:52.556420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:52.556657 (+ 237us) server_negotiation.cc:407] Connection header received
0619 08:59:52.557200 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:52.557201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:52.557215 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:52.557285 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:52.557996 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.558541 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.559929 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.560177 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.560253 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:52.561063 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:52.561064 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:52.561065 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:59:52.561070 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:52.561087 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:59:52.561091 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:52.561137 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:52.561211 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 08:59:52.561238 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 08:59:52.561539 18286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:52.556325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42629 (local address 127.0.98.2:46275)
0619 08:59:52.556531 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:52.556567 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:52.557096 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:52.557366 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:52.557371 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:52.557381 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:59:52.557896 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:52.557902 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.558637 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.558638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:52.559840 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:52.559843 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.560255 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.560256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:52.560403 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:52.560911 (+ 508us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:59:52.560912 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:59:52.560941 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:59:52.560966 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:59:52.561144 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:59:52.561149 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:59:52.561153 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:59:52.561217 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 08:59:52.561323 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":101,"threads_started":1}
I20260619 08:59:52.562008 18284 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42629
I20260619 08:59:52.562093 18284 heartbeater.cc:464] Registering TS with master...
I20260619 08:59:52.562268 18284 heartbeater.cc:511] Master 127.0.98.62:42629 requested a full tablet report, sending...
I20260619 08:59:52.562614 17965 ts_manager.cc:194] Registered new tserver with Master: d6d046ed529a4b09b0923f9fd46095cc (127.0.98.2:36393)
I20260619 08:59:52.562984 17965 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:46275
I20260619 08:59:52.563903 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 18156
I20260619 08:59:52.563982 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-1/wal/instance
I20260619 08:59:52.564199 392 external_mini_cluster.cc:1428] Setting key 282f912c1c4bb71a6b2bc3eac9f92ad5
I20260619 08:59:52.565037 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42629
--builtin_ntp_servers=127.0.98.20:33705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 08:59:52.644269 18287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:59:52.644474 18287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:59:52.644495 18287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:59:52.645658 18287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:59:52.645704 18287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:59:52.645727 18287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 08:59:52.647114 18287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:59:52.647332 18287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:59:52.647614 18287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:59:52.652662 18295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:52.652673 18293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:52.652681 18292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:52.652839 18287 server_base.cc:1034] running on GCE node
I20260619 08:59:52.653067 18287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:59:52.653301 18287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:59:52.654440 18287 hybrid_clock.cc:648] HybridClock initialized: now 1781859592654445 us; error 42 us; skew 500 ppm
I20260619 08:59:52.655586 18287 webserver.cc:460] Webserver started at http://127.0.98.3:33735/ using document root <none> and password file <none>
I20260619 08:59:52.655804 18287 fs_manager.cc:359] Metadata directory not provided
I20260619 08:59:52.655854 18287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:59:52.655992 18287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:59:52.657141 18287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-2/data/instance:
uuid: "46b1d5ddd48845fcb0da4ca9f71f7fd7"
format_stamp: "Formatted at 2026-06-19 08:59:52 on dist-test-slave-h4g5"
server_key: "769b145e9cc696ae54d7496dba7d919b"
server_key_iv: "5c8a74a4d5f0d49cd8c74fa652613593"
server_key_version: "encryptionkey@0"
I20260619 08:59:52.657480 18287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-2/wal/instance:
uuid: "46b1d5ddd48845fcb0da4ca9f71f7fd7"
format_stamp: "Formatted at 2026-06-19 08:59:52 on dist-test-slave-h4g5"
server_key: "769b145e9cc696ae54d7496dba7d919b"
server_key_iv: "5c8a74a4d5f0d49cd8c74fa652613593"
server_key_version: "encryptionkey@0"
I20260619 08:59:52.658735 18287 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 08:59:52.659474 18301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:52.659619 18287 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 08:59:52.659703 18287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-2/wal
uuid: "46b1d5ddd48845fcb0da4ca9f71f7fd7"
format_stamp: "Formatted at 2026-06-19 08:59:52 on dist-test-slave-h4g5"
server_key: "769b145e9cc696ae54d7496dba7d919b"
server_key_iv: "5c8a74a4d5f0d49cd8c74fa652613593"
server_key_version: "encryptionkey@0"
I20260619 08:59:52.659775 18287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:59:52.667026 18287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:59:52.667284 18287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:59:52.667410 18287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:59:52.667618 18287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:59:52.667945 18287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:59:52.667979 18287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:52.668007 18287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:59:52.668025 18287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:52.673331 18287 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45329
I20260619 08:59:52.673396 18414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45329 every 8 connection(s)
I20260619 08:59:52.673699 18287 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
I20260619 08:59:52.678742 18004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:52.674383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58033 (local address 127.0.98.62:42629)
0619 08:59:52.674435 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:52.674437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:52.674725 (+ 288us) server_negotiation.cc:407] Connection header received
0619 08:59:52.675193 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:52.675194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:52.675210 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:52.675257 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:52.675931 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.676483 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.677411 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.677614 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.677676 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:52.678521 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:52.678523 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:52.678523 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:59:52.678527 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:52.678543 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:59:52.678547 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:52.678593 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:52.678655 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 08:59:52.678679 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 08:59:52.678936 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:52.674373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42629 (local address 127.0.98.3:58033)
0619 08:59:52.674607 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:52.674641 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:52.675116 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:52.675321 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:52.675324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:52.675333 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:59:52.675832 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:52.675836 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.676559 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.676560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:52.677339 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:52.677340 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.677686 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.677687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:52.677794 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:52.678358 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:59:52.678359 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:59:52.678404 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:59:52.678430 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:59:52.678598 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:59:52.678601 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:59:52.678603 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:59:52.678650 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 08:59:52.678729 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":128,"threads_started":1}
I20260619 08:59:52.679414 18415 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42629
I20260619 08:59:52.679499 18415 heartbeater.cc:464] Registering TS with master...
I20260619 08:59:52.679678 18415 heartbeater.cc:511] Master 127.0.98.62:42629 requested a full tablet report, sending...
I20260619 08:59:52.679669 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 18287
I20260619 08:59:52.679744 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-2/wal/instance
I20260619 08:59:52.679931 392 external_mini_cluster.cc:1428] Setting key 5cb13e74b6ecbc847efd63479057bbb1
I20260619 08:59:52.680044 17965 ts_manager.cc:194] Registered new tserver with Master: 46b1d5ddd48845fcb0da4ca9f71f7fd7 (127.0.98.3:45329)
I20260619 08:59:52.680534 17965 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:58033
I20260619 08:59:52.680846 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:59:52.684547 18004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:52.682043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36332 (local address 127.0.98.62:42629)
0619 08:59:52.682085 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:52.682087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:52.682183 (+ 96us) server_negotiation.cc:407] Connection header received
0619 08:59:52.682215 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:52.682215 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:52.682223 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:52.682265 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:52.682376 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.682826 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.683514 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.683691 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.683736 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:52.684362 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:52.684363 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:52.684364 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:59:52.684367 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:52.684380 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:59:52.684383 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:52.684422 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:52.684457 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 08:59:52.684476 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 08:59:52.686092 17964 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:59:52.687582 17964 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:59:52.693480 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:52.689582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36393 (local address 127.0.0.1:44384)
0619 08:59:52.689742 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:52.689959 (+ 217us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:52.690010 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:52.690190 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:52.690193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:52.690195 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:59:52.690279 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:52.690281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.691235 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.691236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:52.692418 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:52.692420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.693198 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.693199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:52.693271 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:52.693284 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 08:59:52.693352 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 08:59:52.693403 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":58,"threads_started":1}
I20260619 08:59:52.693491 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:52.689672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44384 (local address 127.0.98.2:36393)
0619 08:59:52.689861 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:52.689866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:52.690017 (+ 151us) server_negotiation.cc:407] Connection header received
0619 08:59:52.690087 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:52.690089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:52.690101 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:52.690182 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:59:52.690338 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.691168 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.692509 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.693126 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.693187 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:52.693276 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:52.693367 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 08:59:52.693411 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":86,"threads_started":1}
I20260619 08:59:52.693961 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:52.689593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35059 (local address 127.0.0.1:52718)
0619 08:59:52.690108 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:52.690116 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:52.690163 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:52.690555 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:52.690555 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:52.690558 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:59:52.690592 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:52.690593 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.691784 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.691785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:52.692674 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:52.692677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.693730 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.693731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:52.693787 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:52.693791 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:59:52.693850 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 08:59:52.693886 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":2,"thread_start_us":86,"threads_started":1}
I20260619 08:59:52.694074 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:52.689817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52718 (local address 127.0.98.1:35059)
0619 08:59:52.689980 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:52.689984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:52.690354 (+ 370us) server_negotiation.cc:407] Connection header received
0619 08:59:52.690407 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:52.690409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:52.690423 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:52.690488 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:59:52.690674 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.691711 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.692748 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.693486 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.693709 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:52.693806 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:52.693869 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 08:59:52.693907 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260619 08:59:52.694271 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:52.689818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39904 (local address 127.0.98.3:45329)
0619 08:59:52.690143 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:52.690148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:52.690157 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:59:52.690193 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:52.690195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:52.690209 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:52.690270 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:59:52.690492 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.691573 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.692773 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.693492 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.693554 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:52.693806 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:52.694151 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 08:59:52.694194 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":76,"threads_started":1}
I20260619 08:59:52.694199 18218 tablet_service.cc:1467] Processing CreateTablet for tablet 4aa57287d71144058ff0c510fef22d9d (DEFAULT_TABLE table=test-table [id=659ee94a34c8490eae9a2b372d74bd6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:59:52.694422 18424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:52.689593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45329 (local address 127.0.0.1:39904)
0619 08:59:52.689914 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:52.689944 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:52.690010 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:52.690277 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:52.690277 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:52.690279 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:59:52.690405 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:52.690408 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.691657 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.691658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:52.692714 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:52.692716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.693840 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.693841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:52.693902 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:52.693906 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:59:52.693955 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 08:59:52.693996 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":98,"threads_started":1}
I20260619 08:59:52.694572 18218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aa57287d71144058ff0c510fef22d9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:59:52.694728 18087 tablet_service.cc:1467] Processing CreateTablet for tablet 4aa57287d71144058ff0c510fef22d9d (DEFAULT_TABLE table=test-table [id=659ee94a34c8490eae9a2b372d74bd6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:59:52.695041 18087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aa57287d71144058ff0c510fef22d9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:59:52.695053 18349 tablet_service.cc:1467] Processing CreateTablet for tablet 4aa57287d71144058ff0c510fef22d9d (DEFAULT_TABLE table=test-table [id=659ee94a34c8490eae9a2b372d74bd6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:59:52.695374 18349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aa57287d71144058ff0c510fef22d9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:59:52.697889 18430 tablet_bootstrap.cc:492] T 4aa57287d71144058ff0c510fef22d9d P d6d046ed529a4b09b0923f9fd46095cc: Bootstrap starting.
I20260619 08:59:52.698154 18431 tablet_bootstrap.cc:492] T 4aa57287d71144058ff0c510fef22d9d P 1c1df67162f243648ff81e2a4aec3911: Bootstrap starting.
I20260619 08:59:52.698199 18429 tablet_bootstrap.cc:492] T 4aa57287d71144058ff0c510fef22d9d P 46b1d5ddd48845fcb0da4ca9f71f7fd7: Bootstrap starting.
I20260619 08:59:52.698689 18429 tablet_bootstrap.cc:654] T 4aa57287d71144058ff0c510fef22d9d P 46b1d5ddd48845fcb0da4ca9f71f7fd7: Neither blocks nor log segments found. Creating new log.
I20260619 08:59:52.698693 18431 tablet_bootstrap.cc:654] T 4aa57287d71144058ff0c510fef22d9d P 1c1df67162f243648ff81e2a4aec3911: Neither blocks nor log segments found. Creating new log.
I20260619 08:59:52.698949 18431 log.cc:824] T 4aa57287d71144058ff0c510fef22d9d P 1c1df67162f243648ff81e2a4aec3911: Log is configured to *not* fsync() on all Append() calls
I20260619 08:59:52.698952 18429 log.cc:824] T 4aa57287d71144058ff0c510fef22d9d P 46b1d5ddd48845fcb0da4ca9f71f7fd7: Log is configured to *not* fsync() on all Append() calls
I20260619 08:59:52.699615 18429 tablet_bootstrap.cc:492] T 4aa57287d71144058ff0c510fef22d9d P 46b1d5ddd48845fcb0da4ca9f71f7fd7: No bootstrap required, opened a new log
I20260619 08:59:52.699621 18431 tablet_bootstrap.cc:492] T 4aa57287d71144058ff0c510fef22d9d P 1c1df67162f243648ff81e2a4aec3911: No bootstrap required, opened a new log
I20260619 08:59:52.699635 18430 tablet_bootstrap.cc:654] T 4aa57287d71144058ff0c510fef22d9d P d6d046ed529a4b09b0923f9fd46095cc: Neither blocks nor log segments found. Creating new log.
I20260619 08:59:52.699683 18431 ts_tablet_manager.cc:1397] T 4aa57287d71144058ff0c510fef22d9d P 1c1df67162f243648ff81e2a4aec3911: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:59:52.699683 18429 ts_tablet_manager.cc:1397] T 4aa57287d71144058ff0c510fef22d9d P 46b1d5ddd48845fcb0da4ca9f71f7fd7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:59:52.699891 18430 log.cc:824] T 4aa57287d71144058ff0c510fef22d9d P d6d046ed529a4b09b0923f9fd46095cc: Log is configured to *not* fsync() on all Append() calls
I20260619 08:59:52.700493 18430 tablet_bootstrap.cc:492] T 4aa57287d71144058ff0c510fef22d9d P d6d046ed529a4b09b0923f9fd46095cc: No bootstrap required, opened a new log
I20260619 08:59:52.700549 18430 ts_tablet_manager.cc:1397] T 4aa57287d71144058ff0c510fef22d9d P d6d046ed529a4b09b0923f9fd46095cc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 08:59:52.702073 18431 raft_consensus.cc:348] T 4aa57287d71144058ff0c510fef22d9d P 1c1df67162f243648ff81e2a4aec3911 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d046ed529a4b09b0923f9fd46095cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "1c1df67162f243648ff81e2a4aec3911" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35059 } } peers { permanent_uuid: "46b1d5ddd48845fcb0da4ca9f71f7fd7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45329 } }
I20260619 08:59:52.702073 18429 raft_consensus.cc:348] T 4aa57287d71144058ff0c510fef22d9d P 46b1d5ddd48845fcb0da4ca9f71f7fd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d046ed529a4b09b0923f9fd46095cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "1c1df67162f243648ff81e2a4aec3911" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35059 } } peers { permanent_uuid: "46b1d5ddd48845fcb0da4ca9f71f7fd7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45329 } }
I20260619 08:59:52.702244 18429 raft_consensus.cc:374] T 4aa57287d71144058ff0c510fef22d9d P 46b1d5ddd48845fcb0da4ca9f71f7fd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:59:52.702244 18431 raft_consensus.cc:374] T 4aa57287d71144058ff0c510fef22d9d P 1c1df67162f243648ff81e2a4aec3911 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:59:52.702278 18431 raft_consensus.cc:729] T 4aa57287d71144058ff0c510fef22d9d P 1c1df67162f243648ff81e2a4aec3911 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c1df67162f243648ff81e2a4aec3911, State: Initialized, Role: FOLLOWER
I20260619 08:59:52.702278 18429 raft_consensus.cc:729] T 4aa57287d71144058ff0c510fef22d9d P 46b1d5ddd48845fcb0da4ca9f71f7fd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46b1d5ddd48845fcb0da4ca9f71f7fd7, State: Initialized, Role: FOLLOWER
I20260619 08:59:52.702416 18429 consensus_queue.cc:260] T 4aa57287d71144058ff0c510fef22d9d P 46b1d5ddd48845fcb0da4ca9f71f7fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d046ed529a4b09b0923f9fd46095cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "1c1df67162f243648ff81e2a4aec3911" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35059 } } peers { permanent_uuid: "46b1d5ddd48845fcb0da4ca9f71f7fd7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45329 } }
I20260619 08:59:52.702447 18431 consensus_queue.cc:260] T 4aa57287d71144058ff0c510fef22d9d P 1c1df67162f243648ff81e2a4aec3911 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d046ed529a4b09b0923f9fd46095cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "1c1df67162f243648ff81e2a4aec3911" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35059 } } peers { permanent_uuid: "46b1d5ddd48845fcb0da4ca9f71f7fd7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45329 } }
I20260619 08:59:52.702529 18430 raft_consensus.cc:348] T 4aa57287d71144058ff0c510fef22d9d P d6d046ed529a4b09b0923f9fd46095cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d046ed529a4b09b0923f9fd46095cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "1c1df67162f243648ff81e2a4aec3911" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35059 } } peers { permanent_uuid: "46b1d5ddd48845fcb0da4ca9f71f7fd7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45329 } }
I20260619 08:59:52.702672 18431 ts_tablet_manager.cc:1428] T 4aa57287d71144058ff0c510fef22d9d P 1c1df67162f243648ff81e2a4aec3911: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:59:52.702672 18429 ts_tablet_manager.cc:1428] T 4aa57287d71144058ff0c510fef22d9d P 46b1d5ddd48845fcb0da4ca9f71f7fd7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:59:52.702679 18430 raft_consensus.cc:374] T 4aa57287d71144058ff0c510fef22d9d P d6d046ed529a4b09b0923f9fd46095cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:59:52.702776 18153 heartbeater.cc:503] Master 127.0.98.62:42629 was elected leader, sending a full tablet report...
I20260619 08:59:52.702788 18415 heartbeater.cc:503] Master 127.0.98.62:42629 was elected leader, sending a full tablet report...
I20260619 08:59:52.702718 18430 raft_consensus.cc:729] T 4aa57287d71144058ff0c510fef22d9d P d6d046ed529a4b09b0923f9fd46095cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6d046ed529a4b09b0923f9fd46095cc, State: Initialized, Role: FOLLOWER
I20260619 08:59:52.703101 18430 consensus_queue.cc:260] T 4aa57287d71144058ff0c510fef22d9d P d6d046ed529a4b09b0923f9fd46095cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d046ed529a4b09b0923f9fd46095cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "1c1df67162f243648ff81e2a4aec3911" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35059 } } peers { permanent_uuid: "46b1d5ddd48845fcb0da4ca9f71f7fd7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45329 } }
I20260619 08:59:52.703358 18430 ts_tablet_manager.cc:1428] T 4aa57287d71144058ff0c510fef22d9d P d6d046ed529a4b09b0923f9fd46095cc: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 08:59:52.703414 18284 heartbeater.cc:503] Master 127.0.98.62:42629 was elected leader, sending a full tablet report...
I20260619 08:59:52.792949 18436 raft_consensus.cc:482] T 4aa57287d71144058ff0c510fef22d9d P 1c1df67162f243648ff81e2a4aec3911 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:59:52.793087 18436 raft_consensus.cc:504] T 4aa57287d71144058ff0c510fef22d9d P 1c1df67162f243648ff81e2a4aec3911 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d046ed529a4b09b0923f9fd46095cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "1c1df67162f243648ff81e2a4aec3911" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35059 } } peers { permanent_uuid: "46b1d5ddd48845fcb0da4ca9f71f7fd7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45329 } }
I20260619 08:59:52.793454 18436 leader_election.cc:288] T 4aa57287d71144058ff0c510fef22d9d P 1c1df67162f243648ff81e2a4aec3911 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6d046ed529a4b09b0923f9fd46095cc (127.0.98.2:36393), 46b1d5ddd48845fcb0da4ca9f71f7fd7 (127.0.98.3:45329)
I20260619 08:59:52.796245 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:52.793742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55223 (local address 127.0.98.2:36393)
0619 08:59:52.793788 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:52.793790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:52.793922 (+ 132us) server_negotiation.cc:407] Connection header received
0619 08:59:52.793951 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:52.793952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:52.793960 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:52.794005 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:59:52.794154 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.794604 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.795566 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.795908 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.795976 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:52.795986 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:52.796141 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 08:59:52.796180 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 08:59:52.796247 18155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:52.793644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45329 (local address 127.0.98.1:39205)
0619 08:59:52.793692 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:52.793704 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:52.793753 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:52.793945 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:52.793945 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:52.793947 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:59:52.794015 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:52.794017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.794652 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.794653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:52.795447 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:52.795449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.795974 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.795975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:52.796042 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:52.796045 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:59:52.796120 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 08:59:52.796166 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":18}
I20260619 08:59:52.796245 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:52.793788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39205 (local address 127.0.98.3:45329)
0619 08:59:52.793835 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:52.793837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:52.793847 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:59:52.793872 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:52.793873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:52.793880 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:52.793942 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:59:52.794088 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.794571 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.795546 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.795892 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.795954 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:52.795964 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:52.796133 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 08:59:52.796175 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 08:59:52.796262 18438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:52.793645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36393 (local address 127.0.98.1:55223)
0619 08:59:52.793840 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:52.793847 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:52.793884 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:52.794021 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:52.794022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:52.794024 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:59:52.794089 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:52.794091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.794686 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.794687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:52.795468 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:52.795470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.795990 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.795991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:52.796047 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:52.796049 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:59:52.796133 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 08:59:52.796175 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":18,"thread_start_us":114,"threads_started":1}
I20260619 08:59:52.796591 18369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aa57287d71144058ff0c510fef22d9d" candidate_uuid: "1c1df67162f243648ff81e2a4aec3911" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46b1d5ddd48845fcb0da4ca9f71f7fd7" is_pre_election: true
I20260619 08:59:52.796592 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aa57287d71144058ff0c510fef22d9d" candidate_uuid: "1c1df67162f243648ff81e2a4aec3911" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6d046ed529a4b09b0923f9fd46095cc" is_pre_election: true
I20260619 08:59:52.796730 18369 raft_consensus.cc:2454] T 4aa57287d71144058ff0c510fef22d9d P 46b1d5ddd48845fcb0da4ca9f71f7fd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c1df67162f243648ff81e2a4aec3911 in term 0.
I20260619 08:59:52.796730 18238 raft_consensus.cc:2454] T 4aa57287d71144058ff0c510fef22d9d P d6d046ed529a4b09b0923f9fd46095cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c1df67162f243648ff81e2a4aec3911 in term 0.
I20260619 08:59:52.796952 18041 leader_election.cc:302] T 4aa57287d71144058ff0c510fef22d9d P 1c1df67162f243648ff81e2a4aec3911 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c1df67162f243648ff81e2a4aec3911, 46b1d5ddd48845fcb0da4ca9f71f7fd7; no voters:
I20260619 08:59:52.797127 18436 raft_consensus.cc:2781] T 4aa57287d71144058ff0c510fef22d9d P 1c1df67162f243648ff81e2a4aec3911 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:59:52.797197 18436 raft_consensus.cc:482] T 4aa57287d71144058ff0c510fef22d9d P 1c1df67162f243648ff81e2a4aec3911 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:59:52.797226 18436 raft_consensus.cc:3037] T 4aa57287d71144058ff0c510fef22d9d P 1c1df67162f243648ff81e2a4aec3911 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:59:52.797999 18436 raft_consensus.cc:504] T 4aa57287d71144058ff0c510fef22d9d P 1c1df67162f243648ff81e2a4aec3911 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d046ed529a4b09b0923f9fd46095cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "1c1df67162f243648ff81e2a4aec3911" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35059 } } peers { permanent_uuid: "46b1d5ddd48845fcb0da4ca9f71f7fd7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45329 } }
I20260619 08:59:52.798146 18436 leader_election.cc:288] T 4aa57287d71144058ff0c510fef22d9d P 1c1df67162f243648ff81e2a4aec3911 [CANDIDATE]: Term 1 election: Requested vote from peers d6d046ed529a4b09b0923f9fd46095cc (127.0.98.2:36393), 46b1d5ddd48845fcb0da4ca9f71f7fd7 (127.0.98.3:45329)
I20260619 08:59:52.798336 18238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aa57287d71144058ff0c510fef22d9d" candidate_uuid: "1c1df67162f243648ff81e2a4aec3911" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6d046ed529a4b09b0923f9fd46095cc"
I20260619 08:59:52.798398 18369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aa57287d71144058ff0c510fef22d9d" candidate_uuid: "1c1df67162f243648ff81e2a4aec3911" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46b1d5ddd48845fcb0da4ca9f71f7fd7"
I20260619 08:59:52.798427 18238 raft_consensus.cc:3037] T 4aa57287d71144058ff0c510fef22d9d P d6d046ed529a4b09b0923f9fd46095cc [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:59:52.798491 18369 raft_consensus.cc:3037] T 4aa57287d71144058ff0c510fef22d9d P 46b1d5ddd48845fcb0da4ca9f71f7fd7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:59:52.799207 18238 raft_consensus.cc:2454] T 4aa57287d71144058ff0c510fef22d9d P d6d046ed529a4b09b0923f9fd46095cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c1df67162f243648ff81e2a4aec3911 in term 1.
I20260619 08:59:52.799206 18369 raft_consensus.cc:2454] T 4aa57287d71144058ff0c510fef22d9d P 46b1d5ddd48845fcb0da4ca9f71f7fd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c1df67162f243648ff81e2a4aec3911 in term 1.
I20260619 08:59:52.799394 18043 leader_election.cc:302] T 4aa57287d71144058ff0c510fef22d9d P 1c1df67162f243648ff81e2a4aec3911 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c1df67162f243648ff81e2a4aec3911, d6d046ed529a4b09b0923f9fd46095cc; no voters:
I20260619 08:59:52.799489 18436 raft_consensus.cc:2781] T 4aa57287d71144058ff0c510fef22d9d P 1c1df67162f243648ff81e2a4aec3911 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:59:52.799654 18436 raft_consensus.cc:686] T 4aa57287d71144058ff0c510fef22d9d P 1c1df67162f243648ff81e2a4aec3911 [term 1 LEADER]: Becoming Leader. State: Replica: 1c1df67162f243648ff81e2a4aec3911, State: Running, Role: LEADER
I20260619 08:59:52.799739 18436 consensus_queue.cc:237] T 4aa57287d71144058ff0c510fef22d9d P 1c1df67162f243648ff81e2a4aec3911 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d046ed529a4b09b0923f9fd46095cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } } peers { permanent_uuid: "1c1df67162f243648ff81e2a4aec3911" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35059 } } peers { permanent_uuid: "46b1d5ddd48845fcb0da4ca9f71f7fd7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45329 } }
I20260619 08:59:52.800616 17964 catalog_manager.cc:5495] T 4aa57287d71144058ff0c510fef22d9d P 1c1df67162f243648ff81e2a4aec3911 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c1df67162f243648ff81e2a4aec3911 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "1c1df67162f243648ff81e2a4aec3911" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d046ed529a4b09b0923f9fd46095cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c1df67162f243648ff81e2a4aec3911" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46b1d5ddd48845fcb0da4ca9f71f7fd7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45329 } health_report { overall_health: UNKNOWN } } }
W20260619 08:59:52.806169 18285 tablet.cc:2369] T 4aa57287d71144058ff0c510fef22d9d P d6d046ed529a4b09b0923f9fd46095cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:59:52.807725 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:52.805540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52728 (local address 127.0.98.1:35059)
0619 08:59:52.805584 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:52.805586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:52.805622 (+ 36us) server_negotiation.cc:407] Connection header received
0619 08:59:52.805649 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:52.805649 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:52.805665 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:52.805715 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:59:52.805818 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.806404 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.807150 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:52.807334 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:52.807387 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:52.807491 (+ 104us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:59:52.807581 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:59:52.807636 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:52.807648 (+ 12us) server_negotiation.cc:299] Negotiation successful
0619 08:59:52.807665 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 08:59:52.809383 18238 raft_consensus.cc:1261] T 4aa57287d71144058ff0c510fef22d9d P d6d046ed529a4b09b0923f9fd46095cc [term 1 FOLLOWER]: Refusing update from remote peer 1c1df67162f243648ff81e2a4aec3911: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:59:52.809387 18369 raft_consensus.cc:1261] T 4aa57287d71144058ff0c510fef22d9d P 46b1d5ddd48845fcb0da4ca9f71f7fd7 [term 1 FOLLOWER]: Refusing update from remote peer 1c1df67162f243648ff81e2a4aec3911: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:59:52.809669 18439 consensus_queue.cc:1035] T 4aa57287d71144058ff0c510fef22d9d P 1c1df67162f243648ff81e2a4aec3911 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46b1d5ddd48845fcb0da4ca9f71f7fd7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:59:52.809784 18436 consensus_queue.cc:1035] T 4aa57287d71144058ff0c510fef22d9d P 1c1df67162f243648ff81e2a4aec3911 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6d046ed529a4b09b0923f9fd46095cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:59:52.812752 18444 mvcc.cc:205] Tried to move back new op lower bound from 7298496892144701440 to 7298496892108136448. Current Snapshot: MvccSnapshot[applied={T|T < 7298496892144701440}]
I20260619 08:59:52.814109 17964 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:59:52.814203 17964 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:59:52.814851 17964 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 143e37781d3d4d3f8521c5d587cce216: Sending DeleteTablet for 3 replicas of tablet 4aa57287d71144058ff0c510fef22d9d
I20260619 08:59:52.815157 18087 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aa57287d71144058ff0c510fef22d9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:59:52 UTC) from {username='slave'} at 127.0.0.1:52718
I20260619 08:59:52.815194 18218 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aa57287d71144058ff0c510fef22d9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:59:52 UTC) from {username='slave'} at 127.0.0.1:44384
I20260619 08:59:52.815212 18349 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aa57287d71144058ff0c510fef22d9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:59:52 UTC) from {username='slave'} at 127.0.0.1:39904
I20260619 08:59:52.815379 18452 tablet_replica.cc:330] T 4aa57287d71144058ff0c510fef22d9d P d6d046ed529a4b09b0923f9fd46095cc: stopping tablet replica
I20260619 08:59:52.815466 18452 raft_consensus.cc:2229] T 4aa57287d71144058ff0c510fef22d9d P d6d046ed529a4b09b0923f9fd46095cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:59:52.815560 18452 raft_consensus.cc:2258] T 4aa57287d71144058ff0c510fef22d9d P d6d046ed529a4b09b0923f9fd46095cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:59:52.815781 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 18025
I20260619 08:59:52.817019 18453 tablet_replica.cc:330] T 4aa57287d71144058ff0c510fef22d9d P 46b1d5ddd48845fcb0da4ca9f71f7fd7: stopping tablet replica
I20260619 08:59:52.817103 18453 raft_consensus.cc:2229] T 4aa57287d71144058ff0c510fef22d9d P 46b1d5ddd48845fcb0da4ca9f71f7fd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:59:52.817200 18453 raft_consensus.cc:2258] T 4aa57287d71144058ff0c510fef22d9d P 46b1d5ddd48845fcb0da4ca9f71f7fd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:59:52.817396 18452 ts_tablet_manager.cc:1905] T 4aa57287d71144058ff0c510fef22d9d P d6d046ed529a4b09b0923f9fd46095cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:59:52.817611 18453 ts_tablet_manager.cc:1905] T 4aa57287d71144058ff0c510fef22d9d P 46b1d5ddd48845fcb0da4ca9f71f7fd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:59:52.819061 18452 ts_tablet_manager.cc:1918] T 4aa57287d71144058ff0c510fef22d9d P d6d046ed529a4b09b0923f9fd46095cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:59:52.819074 18453 ts_tablet_manager.cc:1918] T 4aa57287d71144058ff0c510fef22d9d P 46b1d5ddd48845fcb0da4ca9f71f7fd7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:59:52.819136 18453 log.cc:1196] T 4aa57287d71144058ff0c510fef22d9d P 46b1d5ddd48845fcb0da4ca9f71f7fd7: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-2/wal/wals/4aa57287d71144058ff0c510fef22d9d
I20260619 08:59:52.819136 18452 log.cc:1196] T 4aa57287d71144058ff0c510fef22d9d P d6d046ed529a4b09b0923f9fd46095cc: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859361540516-392-0/minicluster-data/ts-1/wal/wals/4aa57287d71144058ff0c510fef22d9d
I20260619 08:59:52.819373 18452 ts_tablet_manager.cc:1939] T 4aa57287d71144058ff0c510fef22d9d P d6d046ed529a4b09b0923f9fd46095cc: Deleting consensus metadata
I20260619 08:59:52.819373 18453 ts_tablet_manager.cc:1939] T 4aa57287d71144058ff0c510fef22d9d P 46b1d5ddd48845fcb0da4ca9f71f7fd7: Deleting consensus metadata
I20260619 08:59:52.819772 17950 catalog_manager.cc:4841] TS 46b1d5ddd48845fcb0da4ca9f71f7fd7 (127.0.98.3:45329): tablet 4aa57287d71144058ff0c510fef22d9d (table test-table [id=659ee94a34c8490eae9a2b372d74bd6a]) successfully deleted
I20260619 08:59:52.819834 17952 catalog_manager.cc:4841] TS d6d046ed529a4b09b0923f9fd46095cc (127.0.98.2:36393): tablet 4aa57287d71144058ff0c510fef22d9d (table test-table [id=659ee94a34c8490eae9a2b372d74bd6a]) successfully deleted
W20260619 08:59:52.820752 17949 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35059 (error 108)
I20260619 08:59:52.821180 18424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:52.821069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35059 (local address 127.0.0.1:52740)
0619 08:59:52.821120 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:52.821155 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 08:59:52.821249 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 18156
W20260619 08:59:52.821283 17949 catalog_manager.cc:4584] TS 1c1df67162f243648ff81e2a4aec3911 (127.0.98.1:35059): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4aa57287d71144058ff0c510fef22d9d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35059: connect: Connection refused (error 111)
I20260619 08:59:52.825807 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 18287
I20260619 08:59:52.829734 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 17934
2026-06-19T08:59:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (699 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:59:52 dist-test-slave-h4g5 krb5kdc[18457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:59:52 dist-test-slave-h4g5 krb5kdc[18457](info): set up 2 sockets
Jun 19 08:59:52 dist-test-slave-h4g5 krb5kdc[18457](info): commencing operation
krb5kdc: starting...
W20260619 08:59:53.779562 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 08:59:53 dist-test-slave-h4g5 krb5kdc[18457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:59:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:59:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 08:59:53.890877 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43067
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43067
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:59:53.968590 18470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:59:53.968858 18470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:59:53.968883 18470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:59:53.970072 18470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:59:53.970115 18470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:59:53.970129 18470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:59:53.970139 18470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:59:53.970150 18470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:59:53.971612 18470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:59:53.971833 18470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:59:53.972069 18470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:59:53.976758 18476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:53.976791 18478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:53.976774 18475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:53.977094 18470 server_base.cc:1034] running on GCE node
I20260619 08:59:53.977332 18470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:59:53.977634 18470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:59:53.978812 18470 hybrid_clock.cc:648] HybridClock initialized: now 1781859593978799 us; error 31 us; skew 500 ppm
Jun 19 08:59:53 dist-test-slave-h4g5 krb5kdc[18457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:59:53.981328 18470 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 08:59:53.981777 18470 webserver.cc:460] Webserver started at http://127.0.98.62:44209/ using document root <none> and password file <none>
I20260619 08:59:53.981972 18470 fs_manager.cc:359] Metadata directory not provided
I20260619 08:59:53.982019 18470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:59:53.982141 18470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:59:53.983172 18470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "f9804e039ca446228cc22e50fdcd8a7e"
format_stamp: "Formatted at 2026-06-19 08:59:53 on dist-test-slave-h4g5"
server_key: "3e4e58b99e15897f2f673b399e31f0ef"
server_key_iv: "9462c03f270b1417cf598ad054be511c"
server_key_version: "encryptionkey@0"
I20260619 08:59:53.983489 18470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "f9804e039ca446228cc22e50fdcd8a7e"
format_stamp: "Formatted at 2026-06-19 08:59:53 on dist-test-slave-h4g5"
server_key: "3e4e58b99e15897f2f673b399e31f0ef"
server_key_iv: "9462c03f270b1417cf598ad054be511c"
server_key_version: "encryptionkey@0"
I20260619 08:59:53.984803 18470 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:59:53.985565 18485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:53.985736 18470 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:59:53.985816 18470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "f9804e039ca446228cc22e50fdcd8a7e"
format_stamp: "Formatted at 2026-06-19 08:59:53 on dist-test-slave-h4g5"
server_key: "3e4e58b99e15897f2f673b399e31f0ef"
server_key_iv: "9462c03f270b1417cf598ad054be511c"
server_key_version: "encryptionkey@0"
I20260619 08:59:53.985877 18470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:59:54.006505 18470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:59:54.008705 18470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:59:54.008837 18470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:59:54.012637 18470 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43067
I20260619 08:59:54.012707 18537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43067 every 8 connection(s)
I20260619 08:59:54.013192 18470 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:59:54.013876 18538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:59:54.015175 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 18470
I20260619 08:59:54.015267 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 08:59:54.015443 392 external_mini_cluster.cc:1428] Setting key 14647293b43fa355054d1113b41bdac5
I20260619 08:59:54.016350 18538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9804e039ca446228cc22e50fdcd8a7e: Bootstrap starting.
I20260619 08:59:54.017038 18538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9804e039ca446228cc22e50fdcd8a7e: Neither blocks nor log segments found. Creating new log.
I20260619 08:59:54.017313 18538 log.cc:824] T 00000000000000000000000000000000 P f9804e039ca446228cc22e50fdcd8a7e: Log is configured to *not* fsync() on all Append() calls
I20260619 08:59:54.017974 18538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9804e039ca446228cc22e50fdcd8a7e: No bootstrap required, opened a new log
I20260619 08:59:54.020143 18538 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9804e039ca446228cc22e50fdcd8a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9804e039ca446228cc22e50fdcd8a7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43067 } }
I20260619 08:59:54.020320 18538 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9804e039ca446228cc22e50fdcd8a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:59:54.020356 18538 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9804e039ca446228cc22e50fdcd8a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9804e039ca446228cc22e50fdcd8a7e, State: Initialized, Role: FOLLOWER
I20260619 08:59:54.020488 18538 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9804e039ca446228cc22e50fdcd8a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9804e039ca446228cc22e50fdcd8a7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43067 } }
I20260619 08:59:54.020562 18538 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9804e039ca446228cc22e50fdcd8a7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:59:54.020592 18538 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9804e039ca446228cc22e50fdcd8a7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:59:54.020634 18538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9804e039ca446228cc22e50fdcd8a7e [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:59:54.021153 18538 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9804e039ca446228cc22e50fdcd8a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9804e039ca446228cc22e50fdcd8a7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43067 } }
I20260619 08:59:54.021243 18538 leader_election.cc:302] T 00000000000000000000000000000000 P f9804e039ca446228cc22e50fdcd8a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9804e039ca446228cc22e50fdcd8a7e; no voters:
I20260619 08:59:54.021397 18538 leader_election.cc:288] T 00000000000000000000000000000000 P f9804e039ca446228cc22e50fdcd8a7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:59:54.021456 18543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9804e039ca446228cc22e50fdcd8a7e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:59:54.021618 18543 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9804e039ca446228cc22e50fdcd8a7e [term 1 LEADER]: Becoming Leader. State: Replica: f9804e039ca446228cc22e50fdcd8a7e, State: Running, Role: LEADER
I20260619 08:59:54.021646 18538 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9804e039ca446228cc22e50fdcd8a7e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:59:54.021710 18543 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9804e039ca446228cc22e50fdcd8a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9804e039ca446228cc22e50fdcd8a7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43067 } }
I20260619 08:59:54.022054 18543 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9804e039ca446228cc22e50fdcd8a7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9804e039ca446228cc22e50fdcd8a7e. Latest consensus state: current_term: 1 leader_uuid: "f9804e039ca446228cc22e50fdcd8a7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9804e039ca446228cc22e50fdcd8a7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43067 } } }
I20260619 08:59:54.022121 18543 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9804e039ca446228cc22e50fdcd8a7e [sys.catalog]: This master's current role is: LEADER
I20260619 08:59:54.022107 18544 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9804e039ca446228cc22e50fdcd8a7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9804e039ca446228cc22e50fdcd8a7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9804e039ca446228cc22e50fdcd8a7e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43067 } } }
I20260619 08:59:54.022192 18544 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9804e039ca446228cc22e50fdcd8a7e [sys.catalog]: This master's current role is: LEADER
Jun 19 08:59:54 dist-test-slave-h4g5 krb5kdc[18457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:59:54.022346 18551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:59:54.022873 18551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:59:54.024858 18551 catalog_manager.cc:1347] Generated new cluster ID: f5e0931686ba4982b7877dbe2c0c719d
I20260619 08:59:54.024910 18551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:59:54.026137 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:54.015993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37228 (local address 127.0.98.62:43067)
0619 08:59:54.016168 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:54.016177 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:54.016242 (+ 65us) server_negotiation.cc:407] Connection header received
0619 08:59:54.016774 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:54.016783 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:54.016845 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:54.017011 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:54.017598 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.018569 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.019610 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.020002 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.020119 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:54.022820 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:54.022824 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:54.022824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:59:54.022837 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:54.024729 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:54.025081 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:54.025084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:54.025085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:54.025174 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:54.025343 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:54.025344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:54.025344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:54.025580 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:54.025717 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:54.025833 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 08:59:54.025961 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":64,"threads_started":1}
I20260619 08:59:54.032585 18551 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:59:54.033094 18551 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:59:54.041741 18551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9804e039ca446228cc22e50fdcd8a7e: Generated new TSK 0
I20260619 08:59:54.041925 18551 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 08:59:54.067277 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43067
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:59:54.151710 18564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:59:54.151938 18564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:59:54.151962 18564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:59:54.153141 18564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:59:54.153198 18564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:59:54.153225 18564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 08:59:54.154697 18564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:59:54.154911 18564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:59:54.155121 18564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:59:54.159782 18569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:54.159793 18572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:54.159910 18570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:54.159930 18564 server_base.cc:1034] running on GCE node
I20260619 08:59:54.160259 18564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:59:54.160450 18564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:59:54.161614 18564 hybrid_clock.cc:648] HybridClock initialized: now 1781859594161602 us; error 27 us; skew 500 ppm
Jun 19 08:59:54 dist-test-slave-h4g5 krb5kdc[18457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:59:54.164013 18564 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 08:59:54.164469 18564 webserver.cc:460] Webserver started at http://127.0.98.1:36625/ using document root <none> and password file <none>
I20260619 08:59:54.164677 18564 fs_manager.cc:359] Metadata directory not provided
I20260619 08:59:54.164726 18564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:59:54.164844 18564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:59:54.165880 18564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-0/data/instance:
uuid: "1026085540a64197bbbe649d4c283a4d"
format_stamp: "Formatted at 2026-06-19 08:59:54 on dist-test-slave-h4g5"
server_key: "1b31ba2dab358bc4ba823433e6c2c5a3"
server_key_iv: "2b80f9cf066a4bc7ba0aad4d2aced616"
server_key_version: "encryptionkey@0"
I20260619 08:59:54.166198 18564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1026085540a64197bbbe649d4c283a4d"
format_stamp: "Formatted at 2026-06-19 08:59:54 on dist-test-slave-h4g5"
server_key: "1b31ba2dab358bc4ba823433e6c2c5a3"
server_key_iv: "2b80f9cf066a4bc7ba0aad4d2aced616"
server_key_version: "encryptionkey@0"
I20260619 08:59:54.167369 18564 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:59:54.168008 18579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:54.168172 18564 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:59:54.168251 18564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-0/wal
uuid: "1026085540a64197bbbe649d4c283a4d"
format_stamp: "Formatted at 2026-06-19 08:59:54 on dist-test-slave-h4g5"
server_key: "1b31ba2dab358bc4ba823433e6c2c5a3"
server_key_iv: "2b80f9cf066a4bc7ba0aad4d2aced616"
server_key_version: "encryptionkey@0"
I20260619 08:59:54.168314 18564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:59:54.185377 18564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:59:54.187448 18564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:59:54.187587 18564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:59:54.187790 18564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:59:54.188154 18564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:59:54.188196 18564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:54.188232 18564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:59:54.188256 18564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:54.193697 18564 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39995
I20260619 08:59:54.193753 18692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39995 every 8 connection(s)
I20260619 08:59:54.194029 18564 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 08:59:54 dist-test-slave-h4g5 krb5kdc[18457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:59:54.202298 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:54.194645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36913 (local address 127.0.98.62:43067)
0619 08:59:54.194703 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:54.194705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:54.194918 (+ 213us) server_negotiation.cc:407] Connection header received
0619 08:59:54.195851 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:54.195853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:54.195866 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:54.195918 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:54.196529 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.197031 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.197984 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.198186 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.198239 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:54.200074 (+ 1835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:54.200076 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:54.200098 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:59:54.200106 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:54.201128 (+ 1022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:54.201464 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:54.201465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:54.201466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:54.201517 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:54.201723 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:54.201724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:54.201724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:54.201826 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:54.201879 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:54.202124 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 08:59:54.202209 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 08:59:54.202440 18695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:54.194647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43067 (local address 127.0.98.1:36913)
0619 08:59:54.194800 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:54.194834 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:54.195743 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:54.195987 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:54.195991 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:54.196257 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:59:54.196437 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:54.196442 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.197110 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.197111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:54.197906 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:54.197908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.198251 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.198252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:54.198367 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:54.198764 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:59:54.198771 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:59:54.199984 (+ 1213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:59:54.201196 (+ 1212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:59:54.201202 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:59:54.201204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:59:54.201402 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:59:54.201574 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:59:54.201575 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:59:54.201575 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:59:54.201669 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:59:54.201877 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:59:54.201884 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:59:54.202006 (+ 122us) client_negotiation.cc:769] Sending connection context
0619 08:59:54.202093 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 08:59:54.202239 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
I20260619 08:59:54.202510 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 18564
I20260619 08:59:54.202633 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-0/wal/instance
I20260619 08:59:54.202844 392 external_mini_cluster.cc:1428] Setting key 311b9007811fa1ee90a81e19cce8ef89
I20260619 08:59:54.203023 18693 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43067
I20260619 08:59:54.203112 18693 heartbeater.cc:464] Registering TS with master...
I20260619 08:59:54.203274 18693 heartbeater.cc:511] Master 127.0.98.62:43067 requested a full tablet report, sending...
I20260619 08:59:54.203711 18502 ts_manager.cc:194] Registered new tserver with Master: 1026085540a64197bbbe649d4c283a4d (127.0.98.1:39995)
I20260619 08:59:54.204337 18502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36913
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 08:59:54.225098 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43067
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:59:54.303656 18698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:59:54.303855 18698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:59:54.303877 18698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:59:54.305040 18698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:59:54.305116 18698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:59:54.305143 18698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 08:59:54.306631 18698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:59:54.306846 18698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:59:54.307062 18698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:59:54.311672 18704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:54.311798 18698 server_base.cc:1034] running on GCE node
W20260619 08:59:54.311725 18703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:54.311830 18706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:54.312112 18698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:59:54.312326 18698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:59:54.313512 18698 hybrid_clock.cc:648] HybridClock initialized: now 1781859594313495 us; error 29 us; skew 500 ppm
Jun 19 08:59:54 dist-test-slave-h4g5 krb5kdc[18457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:59:54.315907 18698 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 08:59:54.316344 18698 webserver.cc:460] Webserver started at http://127.0.98.2:43431/ using document root <none> and password file <none>
I20260619 08:59:54.316538 18698 fs_manager.cc:359] Metadata directory not provided
I20260619 08:59:54.316614 18698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:59:54.316736 18698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:59:54.317744 18698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-1/data/instance:
uuid: "3788583291f94797b2f5c51d161691ee"
format_stamp: "Formatted at 2026-06-19 08:59:54 on dist-test-slave-h4g5"
server_key: "d1f9ee0a5c0b67666e97da09f2e25122"
server_key_iv: "4cf8b45208f2dfd5979c57883e6265f4"
server_key_version: "encryptionkey@0"
I20260619 08:59:54.318053 18698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3788583291f94797b2f5c51d161691ee"
format_stamp: "Formatted at 2026-06-19 08:59:54 on dist-test-slave-h4g5"
server_key: "d1f9ee0a5c0b67666e97da09f2e25122"
server_key_iv: "4cf8b45208f2dfd5979c57883e6265f4"
server_key_version: "encryptionkey@0"
I20260619 08:59:54.319247 18698 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:59:54.319912 18713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:54.320050 18698 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:59:54.320147 18698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-1/wal
uuid: "3788583291f94797b2f5c51d161691ee"
format_stamp: "Formatted at 2026-06-19 08:59:54 on dist-test-slave-h4g5"
server_key: "d1f9ee0a5c0b67666e97da09f2e25122"
server_key_iv: "4cf8b45208f2dfd5979c57883e6265f4"
server_key_version: "encryptionkey@0"
I20260619 08:59:54.320209 18698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:59:54.339188 18698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:59:54.340955 18698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:59:54.341089 18698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:59:54.341298 18698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:59:54.341589 18698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:59:54.341622 18698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:54.341650 18698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:59:54.341668 18698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:54.347262 18698 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37615
I20260619 08:59:54.347314 18826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37615 every 8 connection(s)
I20260619 08:59:54.347617 18698 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
I20260619 08:59:54.349539 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 18698
I20260619 08:59:54.349651 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-1/wal/instance
I20260619 08:59:54.349879 392 external_mini_cluster.cc:1428] Setting key fbd3c42076214d4c44bdf023d8c87b08
Jun 19 08:59:54 dist-test-slave-h4g5 krb5kdc[18457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 08:59:54.357316 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:54.348280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34519 (local address 127.0.98.62:43067)
0619 08:59:54.348324 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:54.348325 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:54.348528 (+ 203us) server_negotiation.cc:407] Connection header received
0619 08:59:54.349474 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:54.349475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:54.349485 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:54.349538 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:54.350337 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.350912 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.352311 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.352499 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.352556 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:54.354623 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:54.354625 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:54.354625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:59:54.354631 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:54.355889 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:54.356309 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:54.356310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:54.356311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:54.356386 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:54.356692 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:54.356693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:54.356693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:54.356798 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:54.356846 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:54.357160 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 08:59:54.357225 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 08:59:54.357534 18829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:54.348262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43067 (local address 127.0.98.2:34519)
0619 08:59:54.348419 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:54.348461 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:54.349365 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:54.349639 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:54.349644 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:54.350002 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:59:54.350253 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:54.350258 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.351010 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.351011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:54.352228 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:54.352231 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.352569 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.352570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:54.352704 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:54.353185 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:59:54.353204 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:59:54.354526 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:59:54.355963 (+ 1437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:59:54.355969 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:59:54.355972 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:59:54.356246 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:59:54.356467 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:59:54.356468 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:59:54.356469 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:59:54.356623 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:59:54.356858 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:59:54.356865 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:59:54.357030 (+ 165us) client_negotiation.cc:769] Sending connection context
0619 08:59:54.357138 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 08:59:54.357309 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
I20260619 08:59:54.358011 18827 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43067
I20260619 08:59:54.358119 18827 heartbeater.cc:464] Registering TS with master...
I20260619 08:59:54.358289 18827 heartbeater.cc:511] Master 127.0.98.62:43067 requested a full tablet report, sending...
I20260619 08:59:54.358624 18502 ts_manager.cc:194] Registered new tserver with Master: 3788583291f94797b2f5c51d161691ee (127.0.98.2:37615)
I20260619 08:59:54.359007 18502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34519
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 08:59:54.372944 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43067
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:59:54.451784 18832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:59:54.451987 18832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:59:54.452008 18832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:59:54.453137 18832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:59:54.453176 18832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:59:54.453199 18832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 08:59:54.454597 18832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:59:54.454768 18832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:59:54.454938 18832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:59:54.459513 18838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:54.459570 18840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:54.459661 18832 server_base.cc:1034] running on GCE node
W20260619 08:59:54.459512 18837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:54.459920 18832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:59:54.460135 18832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:59:54.461268 18832 hybrid_clock.cc:648] HybridClock initialized: now 1781859594461256 us; error 30 us; skew 500 ppm
Jun 19 08:59:54 dist-test-slave-h4g5 krb5kdc[18457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:59:54.463526 18832 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 08:59:54.463964 18832 webserver.cc:460] Webserver started at http://127.0.98.3:33709/ using document root <none> and password file <none>
I20260619 08:59:54.464185 18832 fs_manager.cc:359] Metadata directory not provided
I20260619 08:59:54.464231 18832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:59:54.464345 18832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:59:54.465312 18832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-2/data/instance:
uuid: "30f3efeaae5840609eee64bf0562057d"
format_stamp: "Formatted at 2026-06-19 08:59:54 on dist-test-slave-h4g5"
server_key: "2932048be7cb1d776f6561bfb832cc7b"
server_key_iv: "257ff14574ab1bc5560c709433c0cc71"
server_key_version: "encryptionkey@0"
I20260619 08:59:54.465622 18832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-2/wal/instance:
uuid: "30f3efeaae5840609eee64bf0562057d"
format_stamp: "Formatted at 2026-06-19 08:59:54 on dist-test-slave-h4g5"
server_key: "2932048be7cb1d776f6561bfb832cc7b"
server_key_iv: "257ff14574ab1bc5560c709433c0cc71"
server_key_version: "encryptionkey@0"
I20260619 08:59:54.466830 18832 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 08:59:54.467514 18847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:54.467648 18832 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 08:59:54.467721 18832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-2/wal
uuid: "30f3efeaae5840609eee64bf0562057d"
format_stamp: "Formatted at 2026-06-19 08:59:54 on dist-test-slave-h4g5"
server_key: "2932048be7cb1d776f6561bfb832cc7b"
server_key_iv: "257ff14574ab1bc5560c709433c0cc71"
server_key_version: "encryptionkey@0"
I20260619 08:59:54.467784 18832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:59:54.481038 18832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:59:54.482652 18832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:59:54.482765 18832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:59:54.482957 18832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:59:54.483232 18832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:59:54.483264 18832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:54.483284 18832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:59:54.483307 18832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:54.488801 18832 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39077
I20260619 08:59:54.488849 18960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39077 every 8 connection(s)
I20260619 08:59:54.489118 18832 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 08:59:54 dist-test-slave-h4g5 krb5kdc[18457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:59:54.497502 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 18832
I20260619 08:59:54.497601 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-2/wal/instance
I20260619 08:59:54.497704 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:54.489703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33943 (local address 127.0.98.62:43067)
0619 08:59:54.489758 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:54.489760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:54.489941 (+ 181us) server_negotiation.cc:407] Connection header received
0619 08:59:54.490841 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:54.490843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:54.490855 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:54.490907 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:54.491493 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.491996 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.492969 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.493175 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.493237 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:54.495182 (+ 1945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:54.495183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:54.495184 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:59:54.495191 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:54.496471 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:54.496852 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:54.496854 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:54.496854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:54.496906 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:54.497110 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:54.497111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:54.497111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:54.497218 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:54.497273 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:54.497533 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 08:59:54.497616 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 08:59:54.497814 392 external_mini_cluster.cc:1428] Setting key 03182ea1cde1375d454f4b959218e651
I20260619 08:59:54.497852 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:54.489679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43067 (local address 127.0.98.3:33943)
0619 08:59:54.489838 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:54.489871 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:54.490745 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:54.490975 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:54.490979 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:54.491232 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:59:54.491410 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:54.491414 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.492075 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.492095 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:54.492889 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:54.492891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.493259 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.493260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:54.493375 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:54.493827 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:59:54.493835 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:59:54.495079 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:59:54.496542 (+ 1463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:59:54.496548 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:59:54.496550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:59:54.496784 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:59:54.496957 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:59:54.496958 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:59:54.496958 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:59:54.497058 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:59:54.497268 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:59:54.497274 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:59:54.497420 (+ 146us) client_negotiation.cc:769] Sending connection context
0619 08:59:54.497504 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 08:59:54.497650 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
I20260619 08:59:54.498251 18961 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43067
I20260619 08:59:54.498344 18961 heartbeater.cc:464] Registering TS with master...
I20260619 08:59:54.498530 18961 heartbeater.cc:511] Master 127.0.98.62:43067 requested a full tablet report, sending...
I20260619 08:59:54.498903 18501 ts_manager.cc:194] Registered new tserver with Master: 30f3efeaae5840609eee64bf0562057d (127.0.98.3:39077)
I20260619 08:59:54.499274 18501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33943
I20260619 08:59:54.509486 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:59:54.516390 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:54.510783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37238 (local address 127.0.98.62:43067)
0619 08:59:54.510826 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:54.510828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:54.510925 (+ 97us) server_negotiation.cc:407] Connection header received
0619 08:59:54.510975 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:54.510976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:54.510985 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:54.511036 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:54.511532 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.511998 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.512749 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.512930 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.512982 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:54.514157 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:54.514158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:54.514159 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:59:54.514166 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:54.515438 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:54.515759 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:54.515759 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:54.515760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:54.515805 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:54.515933 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:54.515933 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:54.515934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:54.516058 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:54.516149 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:54.516251 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 08:59:54.516302 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 08:59:54.517997 18501 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37238:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:59:54.519459 18501 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:59:54.525720 18973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:54.521411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39077 (local address 127.0.0.1:40968)
0619 08:59:54.521736 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:54.521746 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:54.521807 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:54.521966 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:54.521972 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:54.521974 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:59:54.522047 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:54.522049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.523283 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.523283 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:54.524636 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:54.524638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.525428 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.525429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:54.525503 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:54.525514 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 08:59:54.525572 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 08:59:54.525650 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":1,"thread_start_us":64,"threads_started":1}
I20260619 08:59:54.525738 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:54.521507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40968 (local address 127.0.98.3:39077)
0619 08:59:54.521693 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:54.521697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:54.521806 (+ 109us) server_negotiation.cc:407] Connection header received
0619 08:59:54.521870 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:54.521872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:54.521888 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:54.521959 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:59:54.522146 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.523182 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.524702 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.525359 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.525433 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:54.525490 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:54.525602 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 08:59:54.525645 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":75,"threads_started":1}
I20260619 08:59:54.526403 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:54.521583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39995 (local address 127.0.0.1:38716)
0619 08:59:54.521992 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:54.522001 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:54.522054 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:54.522344 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:54.522345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:54.522347 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:59:54.522382 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:54.522384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.523559 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.523560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:54.524555 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:54.524558 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.526181 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.526182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:54.526241 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:54.526246 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 08:59:54.526302 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 08:59:54.526338 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":56,"threads_started":1}
I20260619 08:59:54.526439 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:54.521784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38716 (local address 127.0.98.1:39995)
0619 08:59:54.521952 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:54.521956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:54.522064 (+ 108us) server_negotiation.cc:407] Connection header received
0619 08:59:54.522112 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:54.522114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:54.522130 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:54.522219 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:59:54.522455 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.523471 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.524627 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.525349 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.525433 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:54.525490 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:54.526321 (+ 831us) server_negotiation.cc:299] Negotiation successful
0619 08:59:54.526366 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":75,"threads_started":1}
I20260619 08:59:54.526463 18895 tablet_service.cc:1467] Processing CreateTablet for tablet bba5c0cf38bf4555a87256b02b5fea89 (DEFAULT_TABLE table=test-table [id=075c3a95d3f54e93b82b40d8893f624c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:59:54.526824 18895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bba5c0cf38bf4555a87256b02b5fea89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:59:54.526957 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:54.521506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44252 (local address 127.0.98.2:37615)
0619 08:59:54.521680 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:54.521684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:54.521695 (+ 11us) server_negotiation.cc:407] Connection header received
0619 08:59:54.521787 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:54.521789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:54.521805 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:54.521874 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:59:54.522146 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.523390 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.525525 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.526122 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.526191 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:54.526246 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:54.526837 (+ 591us) server_negotiation.cc:299] Negotiation successful
0619 08:59:54.526883 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":76,"threads_started":1}
I20260619 08:59:54.527128 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:54.521394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37615 (local address 127.0.0.1:44252)
0619 08:59:54.521597 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:54.521636 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:54.521700 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:54.521965 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:54.521968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:54.521971 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:59:54.522064 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:54.522067 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.523481 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.523481 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:54.524633 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:54.524635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.526193 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.526194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:54.526258 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:54.526262 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:59:54.526309 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 08:59:54.526349 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":4,"thread_start_us":55,"threads_started":1}
I20260619 08:59:54.527052 18627 tablet_service.cc:1467] Processing CreateTablet for tablet bba5c0cf38bf4555a87256b02b5fea89 (DEFAULT_TABLE table=test-table [id=075c3a95d3f54e93b82b40d8893f624c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:59:54.527366 18627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bba5c0cf38bf4555a87256b02b5fea89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:59:54.527783 18761 tablet_service.cc:1467] Processing CreateTablet for tablet bba5c0cf38bf4555a87256b02b5fea89 (DEFAULT_TABLE table=test-table [id=075c3a95d3f54e93b82b40d8893f624c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:59:54.528067 18761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bba5c0cf38bf4555a87256b02b5fea89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:59:54.529852 18975 tablet_bootstrap.cc:492] T bba5c0cf38bf4555a87256b02b5fea89 P 1026085540a64197bbbe649d4c283a4d: Bootstrap starting.
I20260619 08:59:54.530131 18976 tablet_bootstrap.cc:492] T bba5c0cf38bf4555a87256b02b5fea89 P 30f3efeaae5840609eee64bf0562057d: Bootstrap starting.
I20260619 08:59:54.530457 18975 tablet_bootstrap.cc:654] T bba5c0cf38bf4555a87256b02b5fea89 P 1026085540a64197bbbe649d4c283a4d: Neither blocks nor log segments found. Creating new log.
I20260619 08:59:54.530719 18976 tablet_bootstrap.cc:654] T bba5c0cf38bf4555a87256b02b5fea89 P 30f3efeaae5840609eee64bf0562057d: Neither blocks nor log segments found. Creating new log.
I20260619 08:59:54.530922 18975 log.cc:824] T bba5c0cf38bf4555a87256b02b5fea89 P 1026085540a64197bbbe649d4c283a4d: Log is configured to *not* fsync() on all Append() calls
I20260619 08:59:54.531133 18977 tablet_bootstrap.cc:492] T bba5c0cf38bf4555a87256b02b5fea89 P 3788583291f94797b2f5c51d161691ee: Bootstrap starting.
I20260619 08:59:54.531283 18976 log.cc:824] T bba5c0cf38bf4555a87256b02b5fea89 P 30f3efeaae5840609eee64bf0562057d: Log is configured to *not* fsync() on all Append() calls
I20260619 08:59:54.531721 18977 tablet_bootstrap.cc:654] T bba5c0cf38bf4555a87256b02b5fea89 P 3788583291f94797b2f5c51d161691ee: Neither blocks nor log segments found. Creating new log.
I20260619 08:59:54.531790 18975 tablet_bootstrap.cc:492] T bba5c0cf38bf4555a87256b02b5fea89 P 1026085540a64197bbbe649d4c283a4d: No bootstrap required, opened a new log
I20260619 08:59:54.531857 18975 ts_tablet_manager.cc:1397] T bba5c0cf38bf4555a87256b02b5fea89 P 1026085540a64197bbbe649d4c283a4d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:59:54.532298 18977 log.cc:824] T bba5c0cf38bf4555a87256b02b5fea89 P 3788583291f94797b2f5c51d161691ee: Log is configured to *not* fsync() on all Append() calls
I20260619 08:59:54.532598 18976 tablet_bootstrap.cc:492] T bba5c0cf38bf4555a87256b02b5fea89 P 30f3efeaae5840609eee64bf0562057d: No bootstrap required, opened a new log
I20260619 08:59:54.532668 18976 ts_tablet_manager.cc:1397] T bba5c0cf38bf4555a87256b02b5fea89 P 30f3efeaae5840609eee64bf0562057d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 08:59:54.532994 18977 tablet_bootstrap.cc:492] T bba5c0cf38bf4555a87256b02b5fea89 P 3788583291f94797b2f5c51d161691ee: No bootstrap required, opened a new log
I20260619 08:59:54.533052 18977 ts_tablet_manager.cc:1397] T bba5c0cf38bf4555a87256b02b5fea89 P 3788583291f94797b2f5c51d161691ee: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:59:54.533658 18975 raft_consensus.cc:348] T bba5c0cf38bf4555a87256b02b5fea89 P 1026085540a64197bbbe649d4c283a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3788583291f94797b2f5c51d161691ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37615 } } peers { permanent_uuid: "30f3efeaae5840609eee64bf0562057d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39077 } } peers { permanent_uuid: "1026085540a64197bbbe649d4c283a4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39995 } }
I20260619 08:59:54.533797 18975 raft_consensus.cc:374] T bba5c0cf38bf4555a87256b02b5fea89 P 1026085540a64197bbbe649d4c283a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:59:54.533826 18975 raft_consensus.cc:729] T bba5c0cf38bf4555a87256b02b5fea89 P 1026085540a64197bbbe649d4c283a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1026085540a64197bbbe649d4c283a4d, State: Initialized, Role: FOLLOWER
I20260619 08:59:54.533959 18975 consensus_queue.cc:260] T bba5c0cf38bf4555a87256b02b5fea89 P 1026085540a64197bbbe649d4c283a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3788583291f94797b2f5c51d161691ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37615 } } peers { permanent_uuid: "30f3efeaae5840609eee64bf0562057d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39077 } } peers { permanent_uuid: "1026085540a64197bbbe649d4c283a4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39995 } }
I20260619 08:59:54.534219 18975 ts_tablet_manager.cc:1428] T bba5c0cf38bf4555a87256b02b5fea89 P 1026085540a64197bbbe649d4c283a4d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 08:59:54.534219 18693 heartbeater.cc:503] Master 127.0.98.62:43067 was elected leader, sending a full tablet report...
I20260619 08:59:54.534526 18976 raft_consensus.cc:348] T bba5c0cf38bf4555a87256b02b5fea89 P 30f3efeaae5840609eee64bf0562057d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3788583291f94797b2f5c51d161691ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37615 } } peers { permanent_uuid: "30f3efeaae5840609eee64bf0562057d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39077 } } peers { permanent_uuid: "1026085540a64197bbbe649d4c283a4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39995 } }
I20260619 08:59:54.534694 18976 raft_consensus.cc:374] T bba5c0cf38bf4555a87256b02b5fea89 P 30f3efeaae5840609eee64bf0562057d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:59:54.534735 18976 raft_consensus.cc:729] T bba5c0cf38bf4555a87256b02b5fea89 P 30f3efeaae5840609eee64bf0562057d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30f3efeaae5840609eee64bf0562057d, State: Initialized, Role: FOLLOWER
I20260619 08:59:54.534860 18976 consensus_queue.cc:260] T bba5c0cf38bf4555a87256b02b5fea89 P 30f3efeaae5840609eee64bf0562057d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3788583291f94797b2f5c51d161691ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37615 } } peers { permanent_uuid: "30f3efeaae5840609eee64bf0562057d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39077 } } peers { permanent_uuid: "1026085540a64197bbbe649d4c283a4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39995 } }
I20260619 08:59:54.534883 18977 raft_consensus.cc:348] T bba5c0cf38bf4555a87256b02b5fea89 P 3788583291f94797b2f5c51d161691ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3788583291f94797b2f5c51d161691ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37615 } } peers { permanent_uuid: "30f3efeaae5840609eee64bf0562057d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39077 } } peers { permanent_uuid: "1026085540a64197bbbe649d4c283a4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39995 } }
I20260619 08:59:54.535033 18977 raft_consensus.cc:374] T bba5c0cf38bf4555a87256b02b5fea89 P 3788583291f94797b2f5c51d161691ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:59:54.535079 18977 raft_consensus.cc:729] T bba5c0cf38bf4555a87256b02b5fea89 P 3788583291f94797b2f5c51d161691ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3788583291f94797b2f5c51d161691ee, State: Initialized, Role: FOLLOWER
I20260619 08:59:54.535148 18976 ts_tablet_manager.cc:1428] T bba5c0cf38bf4555a87256b02b5fea89 P 30f3efeaae5840609eee64bf0562057d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 08:59:54.535248 18961 heartbeater.cc:503] Master 127.0.98.62:43067 was elected leader, sending a full tablet report...
I20260619 08:59:54.535205 18977 consensus_queue.cc:260] T bba5c0cf38bf4555a87256b02b5fea89 P 3788583291f94797b2f5c51d161691ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3788583291f94797b2f5c51d161691ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37615 } } peers { permanent_uuid: "30f3efeaae5840609eee64bf0562057d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39077 } } peers { permanent_uuid: "1026085540a64197bbbe649d4c283a4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39995 } }
I20260619 08:59:54.535458 18977 ts_tablet_manager.cc:1428] T bba5c0cf38bf4555a87256b02b5fea89 P 3788583291f94797b2f5c51d161691ee: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 08:59:54.535470 18827 heartbeater.cc:503] Master 127.0.98.62:43067 was elected leader, sending a full tablet report...
I20260619 08:59:54.552210 18982 raft_consensus.cc:482] T bba5c0cf38bf4555a87256b02b5fea89 P 30f3efeaae5840609eee64bf0562057d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:59:54.552335 18982 raft_consensus.cc:504] T bba5c0cf38bf4555a87256b02b5fea89 P 30f3efeaae5840609eee64bf0562057d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3788583291f94797b2f5c51d161691ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37615 } } peers { permanent_uuid: "30f3efeaae5840609eee64bf0562057d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39077 } } peers { permanent_uuid: "1026085540a64197bbbe649d4c283a4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39995 } }
I20260619 08:59:54.552713 18982 leader_election.cc:288] T bba5c0cf38bf4555a87256b02b5fea89 P 30f3efeaae5840609eee64bf0562057d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3788583291f94797b2f5c51d161691ee (127.0.98.2:37615), 1026085540a64197bbbe649d4c283a4d (127.0.98.1:39995)
I20260619 08:59:54.555301 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:54.552989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34763 (local address 127.0.98.2:37615)
0619 08:59:54.553033 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:54.553035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:54.553042 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:59:54.553067 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:54.553068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:54.553079 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:54.553124 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:59:54.553278 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.553752 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.554642 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.555002 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.555058 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:54.555069 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:54.555200 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 08:59:54.555238 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 08:59:54.555318 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:54.552867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37615 (local address 127.0.98.3:34763)
0619 08:59:54.552922 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:54.552934 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:54.552990 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:54.553132 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:54.553133 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:54.553134 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:59:54.553204 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:54.553205 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.553823 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.553824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:54.554555 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:54.554557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.555065 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.555066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:54.555138 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:54.555141 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:59:54.555185 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 08:59:54.555224 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 08:59:54.555346 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:54.552956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43619 (local address 127.0.98.1:39995)
0619 08:59:54.553014 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:54.553016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:54.553107 (+ 91us) server_negotiation.cc:407] Connection header received
0619 08:59:54.553168 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:54.553170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:54.553180 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:54.553228 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:59:54.553362 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.553815 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.554708 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.555059 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.555117 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:54.555126 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:54.555247 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 08:59:54.555280 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 08:59:54.555413 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:54.552881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39995 (local address 127.0.98.3:43619)
0619 08:59:54.553044 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:54.553053 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:54.553109 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:54.553235 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:54.553236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:54.553237 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:59:54.553309 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:54.553310 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.553885 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.553886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:54.554626 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:54.554628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.555123 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.555124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:54.555191 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:54.555194 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:59:54.555237 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 08:59:54.555272 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":1,"thread_start_us":98,"threads_started":1}
I20260619 08:59:54.555645 18781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bba5c0cf38bf4555a87256b02b5fea89" candidate_uuid: "30f3efeaae5840609eee64bf0562057d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3788583291f94797b2f5c51d161691ee" is_pre_election: true
I20260619 08:59:54.555706 18647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bba5c0cf38bf4555a87256b02b5fea89" candidate_uuid: "30f3efeaae5840609eee64bf0562057d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1026085540a64197bbbe649d4c283a4d" is_pre_election: true
I20260619 08:59:54.555828 18647 raft_consensus.cc:2454] T bba5c0cf38bf4555a87256b02b5fea89 P 1026085540a64197bbbe649d4c283a4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30f3efeaae5840609eee64bf0562057d in term 0.
I20260619 08:59:54.555828 18781 raft_consensus.cc:2454] T bba5c0cf38bf4555a87256b02b5fea89 P 3788583291f94797b2f5c51d161691ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30f3efeaae5840609eee64bf0562057d in term 0.
I20260619 08:59:54.556074 18848 leader_election.cc:302] T bba5c0cf38bf4555a87256b02b5fea89 P 30f3efeaae5840609eee64bf0562057d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1026085540a64197bbbe649d4c283a4d, 30f3efeaae5840609eee64bf0562057d; no voters:
I20260619 08:59:54.556228 18982 raft_consensus.cc:2781] T bba5c0cf38bf4555a87256b02b5fea89 P 30f3efeaae5840609eee64bf0562057d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:59:54.556289 18982 raft_consensus.cc:482] T bba5c0cf38bf4555a87256b02b5fea89 P 30f3efeaae5840609eee64bf0562057d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:59:54.556313 18982 raft_consensus.cc:3037] T bba5c0cf38bf4555a87256b02b5fea89 P 30f3efeaae5840609eee64bf0562057d [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:59:54.557122 18982 raft_consensus.cc:504] T bba5c0cf38bf4555a87256b02b5fea89 P 30f3efeaae5840609eee64bf0562057d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3788583291f94797b2f5c51d161691ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37615 } } peers { permanent_uuid: "30f3efeaae5840609eee64bf0562057d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39077 } } peers { permanent_uuid: "1026085540a64197bbbe649d4c283a4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39995 } }
I20260619 08:59:54.557235 18982 leader_election.cc:288] T bba5c0cf38bf4555a87256b02b5fea89 P 30f3efeaae5840609eee64bf0562057d [CANDIDATE]: Term 1 election: Requested vote from peers 3788583291f94797b2f5c51d161691ee (127.0.98.2:37615), 1026085540a64197bbbe649d4c283a4d (127.0.98.1:39995)
I20260619 08:59:54.557385 18781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bba5c0cf38bf4555a87256b02b5fea89" candidate_uuid: "30f3efeaae5840609eee64bf0562057d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3788583291f94797b2f5c51d161691ee"
I20260619 08:59:54.557428 18647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bba5c0cf38bf4555a87256b02b5fea89" candidate_uuid: "30f3efeaae5840609eee64bf0562057d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1026085540a64197bbbe649d4c283a4d"
I20260619 08:59:54.557469 18781 raft_consensus.cc:3037] T bba5c0cf38bf4555a87256b02b5fea89 P 3788583291f94797b2f5c51d161691ee [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:59:54.557494 18647 raft_consensus.cc:3037] T bba5c0cf38bf4555a87256b02b5fea89 P 1026085540a64197bbbe649d4c283a4d [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:59:54.558151 18647 raft_consensus.cc:2454] T bba5c0cf38bf4555a87256b02b5fea89 P 1026085540a64197bbbe649d4c283a4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30f3efeaae5840609eee64bf0562057d in term 1.
I20260619 08:59:54.558204 18781 raft_consensus.cc:2454] T bba5c0cf38bf4555a87256b02b5fea89 P 3788583291f94797b2f5c51d161691ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30f3efeaae5840609eee64bf0562057d in term 1.
I20260619 08:59:54.558334 18848 leader_election.cc:302] T bba5c0cf38bf4555a87256b02b5fea89 P 30f3efeaae5840609eee64bf0562057d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1026085540a64197bbbe649d4c283a4d, 30f3efeaae5840609eee64bf0562057d, 3788583291f94797b2f5c51d161691ee; no voters:
I20260619 08:59:54.558427 18982 raft_consensus.cc:2781] T bba5c0cf38bf4555a87256b02b5fea89 P 30f3efeaae5840609eee64bf0562057d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:59:54.558543 18982 raft_consensus.cc:686] T bba5c0cf38bf4555a87256b02b5fea89 P 30f3efeaae5840609eee64bf0562057d [term 1 LEADER]: Becoming Leader. State: Replica: 30f3efeaae5840609eee64bf0562057d, State: Running, Role: LEADER
I20260619 08:59:54.558620 18982 consensus_queue.cc:237] T bba5c0cf38bf4555a87256b02b5fea89 P 30f3efeaae5840609eee64bf0562057d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3788583291f94797b2f5c51d161691ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37615 } } peers { permanent_uuid: "30f3efeaae5840609eee64bf0562057d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39077 } } peers { permanent_uuid: "1026085540a64197bbbe649d4c283a4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39995 } }
I20260619 08:59:54.559388 18501 catalog_manager.cc:5495] T bba5c0cf38bf4555a87256b02b5fea89 P 30f3efeaae5840609eee64bf0562057d reported cstate change: term changed from 0 to 1, leader changed from <none> to 30f3efeaae5840609eee64bf0562057d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "30f3efeaae5840609eee64bf0562057d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3788583291f94797b2f5c51d161691ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30f3efeaae5840609eee64bf0562057d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1026085540a64197bbbe649d4c283a4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39995 } health_report { overall_health: UNKNOWN } } }
I20260619 08:59:54.570845 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:54.568490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40984 (local address 127.0.98.3:39077)
0619 08:59:54.568578 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:54.568580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:54.568588 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:59:54.568621 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:54.568623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:54.568640 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:54.568700 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:59:54.568818 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.569276 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.570033 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.570273 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.570321 (+ 48us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:54.570508 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:59:54.570612 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:59:54.570696 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:54.570741 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 08:59:54.570747 (+ 6us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 08:59:54.572670 18647 raft_consensus.cc:1261] T bba5c0cf38bf4555a87256b02b5fea89 P 1026085540a64197bbbe649d4c283a4d [term 1 FOLLOWER]: Refusing update from remote peer 30f3efeaae5840609eee64bf0562057d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:59:54.572754 18781 raft_consensus.cc:1261] T bba5c0cf38bf4555a87256b02b5fea89 P 3788583291f94797b2f5c51d161691ee [term 1 FOLLOWER]: Refusing update from remote peer 30f3efeaae5840609eee64bf0562057d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:59:54.572981 18985 consensus_queue.cc:1035] T bba5c0cf38bf4555a87256b02b5fea89 P 30f3efeaae5840609eee64bf0562057d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1026085540a64197bbbe649d4c283a4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:59:54.573114 18982 consensus_queue.cc:1035] T bba5c0cf38bf4555a87256b02b5fea89 P 30f3efeaae5840609eee64bf0562057d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3788583291f94797b2f5c51d161691ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:59:54.575068 18988 mvcc.cc:205] Tried to move back new op lower bound from 7298496899366862848 to 7298496899312676864. Current Snapshot: MvccSnapshot[applied={T|T < 7298496899366862848}]
I20260619 08:59:54.575443 18990 mvcc.cc:205] Tried to move back new op lower bound from 7298496899366862848 to 7298496899312676864. Current Snapshot: MvccSnapshot[applied={T|T < 7298496899366862848}]
I20260619 08:59:54.577839 18502 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:59:54.577919 18502 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:59:54.578593 18502 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f9804e039ca446228cc22e50fdcd8a7e: Sending DeleteTablet for 3 replicas of tablet bba5c0cf38bf4555a87256b02b5fea89
I20260619 08:59:54.578882 18761 tablet_service.cc:1514] Processing DeleteTablet for tablet bba5c0cf38bf4555a87256b02b5fea89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:59:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44252
I20260619 08:59:54.578948 18627 tablet_service.cc:1514] Processing DeleteTablet for tablet bba5c0cf38bf4555a87256b02b5fea89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:59:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38716
I20260619 08:59:54.579228 18895 tablet_service.cc:1514] Processing DeleteTablet for tablet bba5c0cf38bf4555a87256b02b5fea89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:59:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40968
I20260619 08:59:54.579278 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 18564
I20260619 08:59:54.579819 18998 tablet_replica.cc:330] T bba5c0cf38bf4555a87256b02b5fea89 P 3788583291f94797b2f5c51d161691ee: stopping tablet replica
I20260619 08:59:54.579911 18998 raft_consensus.cc:2229] T bba5c0cf38bf4555a87256b02b5fea89 P 3788583291f94797b2f5c51d161691ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:59:54.580003 18998 raft_consensus.cc:2258] T bba5c0cf38bf4555a87256b02b5fea89 P 3788583291f94797b2f5c51d161691ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:59:54.580762 18998 ts_tablet_manager.cc:1905] T bba5c0cf38bf4555a87256b02b5fea89 P 3788583291f94797b2f5c51d161691ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:59:54.581856 19000 tablet_replica.cc:330] T bba5c0cf38bf4555a87256b02b5fea89 P 30f3efeaae5840609eee64bf0562057d: stopping tablet replica
I20260619 08:59:54.581961 19000 raft_consensus.cc:2229] T bba5c0cf38bf4555a87256b02b5fea89 P 30f3efeaae5840609eee64bf0562057d [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:59:54.582094 19000 raft_consensus.cc:2258] T bba5c0cf38bf4555a87256b02b5fea89 P 30f3efeaae5840609eee64bf0562057d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:59:54.582122 18998 ts_tablet_manager.cc:1918] T bba5c0cf38bf4555a87256b02b5fea89 P 3788583291f94797b2f5c51d161691ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:59:54.582181 18998 log.cc:1196] T bba5c0cf38bf4555a87256b02b5fea89 P 3788583291f94797b2f5c51d161691ee: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-1/wal/wals/bba5c0cf38bf4555a87256b02b5fea89
I20260619 08:59:54.582504 19000 ts_tablet_manager.cc:1905] T bba5c0cf38bf4555a87256b02b5fea89 P 30f3efeaae5840609eee64bf0562057d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:59:54.582592 18998 ts_tablet_manager.cc:1939] T bba5c0cf38bf4555a87256b02b5fea89 P 3788583291f94797b2f5c51d161691ee: Deleting consensus metadata
I20260619 08:59:54.582965 18487 catalog_manager.cc:4841] TS 3788583291f94797b2f5c51d161691ee (127.0.98.2:37615): tablet bba5c0cf38bf4555a87256b02b5fea89 (table test-table [id=075c3a95d3f54e93b82b40d8893f624c]) successfully deleted
I20260619 08:59:54.584028 19000 ts_tablet_manager.cc:1918] T bba5c0cf38bf4555a87256b02b5fea89 P 30f3efeaae5840609eee64bf0562057d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:59:54.584120 19000 log.cc:1196] T bba5c0cf38bf4555a87256b02b5fea89 P 30f3efeaae5840609eee64bf0562057d: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859361540516-392-0/minicluster-data/ts-2/wal/wals/bba5c0cf38bf4555a87256b02b5fea89
I20260619 08:59:54.584347 19000 ts_tablet_manager.cc:1939] T bba5c0cf38bf4555a87256b02b5fea89 P 30f3efeaae5840609eee64bf0562057d: Deleting consensus metadata
I20260619 08:59:54.584676 18486 catalog_manager.cc:4841] TS 30f3efeaae5840609eee64bf0562057d (127.0.98.3:39077): tablet bba5c0cf38bf4555a87256b02b5fea89 (table test-table [id=075c3a95d3f54e93b82b40d8893f624c]) successfully deleted
W20260619 08:59:54.585927 18486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39995 (error 108)
I20260619 08:59:54.586182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 18698
I20260619 08:59:54.586395 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:54.586268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39995 (local address 127.0.0.1:38728)
0619 08:59:54.586321 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:54.586361 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 08:59:54.586511 18486 catalog_manager.cc:4584] TS 1026085540a64197bbbe649d4c283a4d (127.0.98.1:39995): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bba5c0cf38bf4555a87256b02b5fea89: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39995: connect: Connection refused (error 111)
I20260619 08:59:54.591305 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 18832
I20260619 08:59:54.595503 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 18470
2026-06-19T08:59:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1766 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T08:59:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:59:54Z Disabled control of system clock
I20260619 08:59:54.620949 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46109
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 08:59:54.698261 19005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:59:54.698473 19005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:59:54.698493 19005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:59:54.699584 19005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:59:54.699618 19005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:59:54.699638 19005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:59:54.699649 19005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:59:54.699659 19005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:59:54.700966 19005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46109
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:59:54.701155 19005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:59:54.701392 19005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:59:54.705850 19011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:54.705932 19010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:54.706079 19005 server_base.cc:1034] running on GCE node
W20260619 08:59:54.706120 19013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:54.706321 19005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:59:54.706583 19005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:59:54.707703 19005 hybrid_clock.cc:648] HybridClock initialized: now 1781859594707677 us; error 26 us; skew 500 ppm
I20260619 08:59:54.708910 19005 webserver.cc:460] Webserver started at http://127.0.98.62:43669/ using document root <none> and password file <none>
I20260619 08:59:54.709138 19005 fs_manager.cc:359] Metadata directory not provided
I20260619 08:59:54.709182 19005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:59:54.709290 19005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:59:54.710300 19005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "721abaf638d447f59e019d36ef073187"
format_stamp: "Formatted at 2026-06-19 08:59:54 on dist-test-slave-h4g5"
server_key: "943ba4f1bce2a5f099ca3bdab38ad542"
server_key_iv: "f40aa232fe9cdbded95f9112cabdae60"
server_key_version: "encryptionkey@0"
I20260619 08:59:54.710615 19005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "721abaf638d447f59e019d36ef073187"
format_stamp: "Formatted at 2026-06-19 08:59:54 on dist-test-slave-h4g5"
server_key: "943ba4f1bce2a5f099ca3bdab38ad542"
server_key_iv: "f40aa232fe9cdbded95f9112cabdae60"
server_key_version: "encryptionkey@0"
I20260619 08:59:54.711787 19005 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:59:54.712458 19019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:54.712605 19005 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:54.712682 19005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "721abaf638d447f59e019d36ef073187"
format_stamp: "Formatted at 2026-06-19 08:59:54 on dist-test-slave-h4g5"
server_key: "943ba4f1bce2a5f099ca3bdab38ad542"
server_key_iv: "f40aa232fe9cdbded95f9112cabdae60"
server_key_version: "encryptionkey@0"
I20260619 08:59:54.712766 19005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:59:54.735689 19005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:59:54.736025 19005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:59:54.736181 19005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:59:54.739705 19005 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46109
I20260619 08:59:54.739745 19071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46109 every 8 connection(s)
I20260619 08:59:54.740041 19005 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:59:54.740968 19072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:59:54.743196 19072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 721abaf638d447f59e019d36ef073187: Bootstrap starting.
I20260619 08:59:54.743734 19072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 721abaf638d447f59e019d36ef073187: Neither blocks nor log segments found. Creating new log.
I20260619 08:59:54.744001 19072 log.cc:824] T 00000000000000000000000000000000 P 721abaf638d447f59e019d36ef073187: Log is configured to *not* fsync() on all Append() calls
I20260619 08:59:54.744704 19072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 721abaf638d447f59e019d36ef073187: No bootstrap required, opened a new log
I20260619 08:59:54.744825 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 19005
I20260619 08:59:54.744936 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 08:59:54.745131 392 external_mini_cluster.cc:1428] Setting key be118edb96c88fdab3e011f099a0ff68
I20260619 08:59:54.746635 19072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 721abaf638d447f59e019d36ef073187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "721abaf638d447f59e019d36ef073187" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46109 } }
I20260619 08:59:54.746773 19072 raft_consensus.cc:374] T 00000000000000000000000000000000 P 721abaf638d447f59e019d36ef073187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:59:54.746805 19072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 721abaf638d447f59e019d36ef073187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721abaf638d447f59e019d36ef073187, State: Initialized, Role: FOLLOWER
I20260619 08:59:54.746933 19072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 721abaf638d447f59e019d36ef073187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "721abaf638d447f59e019d36ef073187" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46109 } }
I20260619 08:59:54.747009 19072 raft_consensus.cc:388] T 00000000000000000000000000000000 P 721abaf638d447f59e019d36ef073187 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:59:54.747051 19072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 721abaf638d447f59e019d36ef073187 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:59:54.747095 19072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 721abaf638d447f59e019d36ef073187 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:59:54.747591 19072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 721abaf638d447f59e019d36ef073187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "721abaf638d447f59e019d36ef073187" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46109 } }
I20260619 08:59:54.747676 19072 leader_election.cc:302] T 00000000000000000000000000000000 P 721abaf638d447f59e019d36ef073187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 721abaf638d447f59e019d36ef073187; no voters:
I20260619 08:59:54.747833 19072 leader_election.cc:288] T 00000000000000000000000000000000 P 721abaf638d447f59e019d36ef073187 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:59:54.747864 19077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 721abaf638d447f59e019d36ef073187 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:59:54.747972 19077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 721abaf638d447f59e019d36ef073187 [term 1 LEADER]: Becoming Leader. State: Replica: 721abaf638d447f59e019d36ef073187, State: Running, Role: LEADER
I20260619 08:59:54.748099 19072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 721abaf638d447f59e019d36ef073187 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:59:54.748049 19077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 721abaf638d447f59e019d36ef073187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "721abaf638d447f59e019d36ef073187" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46109 } }
I20260619 08:59:54.748446 19077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 721abaf638d447f59e019d36ef073187 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 721abaf638d447f59e019d36ef073187. Latest consensus state: current_term: 1 leader_uuid: "721abaf638d447f59e019d36ef073187" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "721abaf638d447f59e019d36ef073187" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46109 } } }
I20260619 08:59:54.748458 19079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 721abaf638d447f59e019d36ef073187 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "721abaf638d447f59e019d36ef073187" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "721abaf638d447f59e019d36ef073187" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46109 } } }
I20260619 08:59:54.748522 19077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 721abaf638d447f59e019d36ef073187 [sys.catalog]: This master's current role is: LEADER
I20260619 08:59:54.748551 19079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 721abaf638d447f59e019d36ef073187 [sys.catalog]: This master's current role is: LEADER
W20260619 08:59:54.749977 19093 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 721abaf638d447f59e019d36ef073187: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 08:59:54.750030 19093 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 08:59:54.750069 19081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:59:54.750212 19081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:59:54.751696 19081 catalog_manager.cc:1347] Generated new cluster ID: 4d399d680ec74c6994e920a527e9ba18
I20260619 08:59:54.751753 19081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:59:54.751893 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:54.745630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47656 (local address 127.0.98.62:46109)
0619 08:59:54.745809 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:54.745817 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:54.745840 (+ 23us) server_negotiation.cc:407] Connection header received
0619 08:59:54.746604 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:54.746610 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:54.746666 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:54.746793 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:54.747053 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.748765 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.749728 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.750227 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.750369 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:54.751086 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:54.751090 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:54.751090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:59:54.751283 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:54.751331 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:59:54.751347 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:54.751408 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:54.751467 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 08:59:54.751634 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":57,"threads_started":1}
I20260619 08:59:54.765525 19081 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:59:54.766263 19081 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:59:54.770982 19081 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 721abaf638d447f59e019d36ef073187: Generated new TSK 0
I20260619 08:59:54.771171 19081 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:59:54.781744 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46109
--builtin_ntp_servers=127.0.98.20:41497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 08:59:54.859826 19096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:59:54.860033 19096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:59:54.860061 19096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:59:54.861213 19096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:59:54.861253 19096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:59:54.861284 19096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 08:59:54.862581 19096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:59:54.862821 19096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:59:54.863085 19096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:59:54.867614 19102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:54.867657 19101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:54.867614 19104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:54.867879 19096 server_base.cc:1034] running on GCE node
I20260619 08:59:54.868064 19096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:59:54.868306 19096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:59:54.869477 19096 hybrid_clock.cc:648] HybridClock initialized: now 1781859594869460 us; error 31 us; skew 500 ppm
I20260619 08:59:54.870836 19096 webserver.cc:460] Webserver started at http://127.0.98.1:38985/ using document root <none> and password file <none>
I20260619 08:59:54.871047 19096 fs_manager.cc:359] Metadata directory not provided
I20260619 08:59:54.871091 19096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:59:54.871219 19096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:59:54.872282 19096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-0/data/instance:
uuid: "f3d7e04437e24cf68be5bbea6fe9f7cf"
format_stamp: "Formatted at 2026-06-19 08:59:54 on dist-test-slave-h4g5"
server_key: "d05a0bcaa4a34373174806a6c37060b4"
server_key_iv: "2bc1970b4a40d9f104f2c060b96cffc4"
server_key_version: "encryptionkey@0"
I20260619 08:59:54.872601 19096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f3d7e04437e24cf68be5bbea6fe9f7cf"
format_stamp: "Formatted at 2026-06-19 08:59:54 on dist-test-slave-h4g5"
server_key: "d05a0bcaa4a34373174806a6c37060b4"
server_key_iv: "2bc1970b4a40d9f104f2c060b96cffc4"
server_key_version: "encryptionkey@0"
I20260619 08:59:54.873797 19096 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:59:54.874430 19110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:54.874565 19096 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:59:54.874650 19096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-0/wal
uuid: "f3d7e04437e24cf68be5bbea6fe9f7cf"
format_stamp: "Formatted at 2026-06-19 08:59:54 on dist-test-slave-h4g5"
server_key: "d05a0bcaa4a34373174806a6c37060b4"
server_key_iv: "2bc1970b4a40d9f104f2c060b96cffc4"
server_key_version: "encryptionkey@0"
I20260619 08:59:54.874711 19096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:59:54.880949 19096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:59:54.881192 19096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:59:54.881310 19096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:59:54.881498 19096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:59:54.881786 19096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:59:54.881819 19096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:54.881845 19096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:59:54.881868 19096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:54.887199 19096 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38277
I20260619 08:59:54.887250 19223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38277 every 8 connection(s)
I20260619 08:59:54.887529 19096 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
I20260619 08:59:54.892359 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:54.888171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43687 (local address 127.0.98.62:46109)
0619 08:59:54.888214 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:54.888216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:54.888454 (+ 238us) server_negotiation.cc:407] Connection header received
0619 08:59:54.888949 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:54.888950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:54.888965 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:54.889016 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:54.889650 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.890157 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.891081 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.891277 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.891331 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:54.892036 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:54.892037 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:54.892037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:59:54.892041 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:54.892057 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:59:54.892061 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:54.892180 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:54.892252 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 08:59:54.892280 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 08:59:54.892526 19226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:54.888166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46109 (local address 127.0.98.1:43687)
0619 08:59:54.888340 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:54.888373 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:54.888857 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:54.889084 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:54.889088 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:54.889106 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:59:54.889569 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:54.889574 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.890239 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.890241 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:54.891003 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:54.891005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:54.891349 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:54.891350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:54.891469 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:54.891917 (+ 448us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:59:54.891917 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:59:54.891943 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:59:54.891963 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:59:54.892182 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:59:54.892188 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:59:54.892191 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:59:54.892250 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 08:59:54.892347 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260619 08:59:54.892952 19224 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46109
I20260619 08:59:54.893039 19224 heartbeater.cc:464] Registering TS with master...
I20260619 08:59:54.893213 19224 heartbeater.cc:511] Master 127.0.98.62:46109 requested a full tablet report, sending...
I20260619 08:59:54.893692 19036 ts_manager.cc:194] Registered new tserver with Master: f3d7e04437e24cf68be5bbea6fe9f7cf (127.0.98.1:38277)
I20260619 08:59:54.894316 19036 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:43687
I20260619 08:59:54.896556 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 19096
I20260619 08:59:54.896667 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-0/wal/instance
I20260619 08:59:54.896883 392 external_mini_cluster.cc:1428] Setting key fa7021e08e8969593d622c8ce95a4a9e
I20260619 08:59:54.897821 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46109
--builtin_ntp_servers=127.0.98.20:41497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 08:59:54.976121 19227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:59:54.976334 19227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:59:54.976363 19227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:59:54.977560 19227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:59:54.977597 19227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:59:54.977630 19227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 08:59:54.979014 19227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:59:54.979254 19227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:59:54.979527 19227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:59:54.984229 19232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:54.984259 19233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:54.984230 19235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:54.984491 19227 server_base.cc:1034] running on GCE node
I20260619 08:59:54.984730 19227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:59:54.984951 19227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:59:54.986089 19227 hybrid_clock.cc:648] HybridClock initialized: now 1781859594986049 us; error 51 us; skew 500 ppm
I20260619 08:59:54.987313 19227 webserver.cc:460] Webserver started at http://127.0.98.2:41723/ using document root <none> and password file <none>
I20260619 08:59:54.987540 19227 fs_manager.cc:359] Metadata directory not provided
I20260619 08:59:54.987587 19227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:59:54.987708 19227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:59:54.988790 19227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-1/data/instance:
uuid: "7db7750599754a1381a7e4469b99f684"
format_stamp: "Formatted at 2026-06-19 08:59:54 on dist-test-slave-h4g5"
server_key: "6a7d021b79b0231cc92b160c814f07fb"
server_key_iv: "9398436c81e72a97834588f7c4f18b9e"
server_key_version: "encryptionkey@0"
I20260619 08:59:54.989109 19227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7db7750599754a1381a7e4469b99f684"
format_stamp: "Formatted at 2026-06-19 08:59:54 on dist-test-slave-h4g5"
server_key: "6a7d021b79b0231cc92b160c814f07fb"
server_key_iv: "9398436c81e72a97834588f7c4f18b9e"
server_key_version: "encryptionkey@0"
I20260619 08:59:54.990321 19227 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:59:54.991060 19241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:54.991293 19227 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:59:54.991380 19227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-1/wal
uuid: "7db7750599754a1381a7e4469b99f684"
format_stamp: "Formatted at 2026-06-19 08:59:54 on dist-test-slave-h4g5"
server_key: "6a7d021b79b0231cc92b160c814f07fb"
server_key_iv: "9398436c81e72a97834588f7c4f18b9e"
server_key_version: "encryptionkey@0"
I20260619 08:59:54.991442 19227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:59:55.001451 19227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:59:55.001721 19227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:59:55.001848 19227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:59:55.002043 19227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:59:55.002345 19227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:59:55.002378 19227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:55.002406 19227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:59:55.002424 19227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:55.007778 19227 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46529
I20260619 08:59:55.007827 19354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46529 every 8 connection(s)
I20260619 08:59:55.008213 19227 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
I20260619 08:59:55.012056 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 19227
I20260619 08:59:55.012171 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-1/wal/instance
I20260619 08:59:55.012390 392 external_mini_cluster.cc:1428] Setting key 40572831539a0936e3013c26ab652dd1
I20260619 08:59:55.013407 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:55.008900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48455 (local address 127.0.98.62:46109)
0619 08:59:55.008948 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:55.008949 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:55.009182 (+ 233us) server_negotiation.cc:407] Connection header received
0619 08:59:55.009698 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:55.009700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:55.009713 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:55.009759 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:55.010431 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:55.010959 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:55.011920 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:55.012157 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:55.012219 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:55.013136 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:55.013138 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:55.013138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:59:55.013143 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:55.013162 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:59:55.013167 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:55.013222 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:55.013324 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 08:59:55.013347 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 08:59:55.013449 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46109
--builtin_ntp_servers=127.0.98.20:41497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 08:59:55.013649 19357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:55.008873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46109 (local address 127.0.98.2:48455)
0619 08:59:55.009042 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:55.009087 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:55.009588 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:55.009838 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:55.009842 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:55.009852 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:59:55.010342 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:55.010346 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:55.011040 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:55.011041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:55.011832 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:55.011834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:55.012253 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:55.012255 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:55.012422 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:55.012979 (+ 557us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:59:55.012979 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:59:55.013022 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:59:55.013056 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:59:55.013233 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:59:55.013239 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:59:55.013242 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:59:55.013328 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 08:59:55.013435 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
I20260619 08:59:55.014190 19355 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46109
I20260619 08:59:55.014292 19355 heartbeater.cc:464] Registering TS with master...
I20260619 08:59:55.014467 19355 heartbeater.cc:511] Master 127.0.98.62:46109 requested a full tablet report, sending...
I20260619 08:59:55.014829 19036 ts_manager.cc:194] Registered new tserver with Master: 7db7750599754a1381a7e4469b99f684 (127.0.98.2:46529)
I20260619 08:59:55.015182 19036 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:48455
W20260619 08:59:55.093885 19358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:59:55.094072 19358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:59:55.094092 19358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:59:55.095242 19358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:59:55.095285 19358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:59:55.095310 19358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 08:59:55.096740 19358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:59:55.096967 19358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:59:55.097232 19358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 08:59:55.102082 19358 server_base.cc:1034] running on GCE node
W20260619 08:59:55.102058 19364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:55.102058 19363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:55.102052 19366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:55.102435 19358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:59:55.102649 19358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:59:55.103792 19358 hybrid_clock.cc:648] HybridClock initialized: now 1781859595103764 us; error 44 us; skew 500 ppm
I20260619 08:59:55.105000 19358 webserver.cc:460] Webserver started at http://127.0.98.3:40725/ using document root <none> and password file <none>
I20260619 08:59:55.105213 19358 fs_manager.cc:359] Metadata directory not provided
I20260619 08:59:55.105262 19358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:59:55.105386 19358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:59:55.106459 19358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-2/data/instance:
uuid: "0b12ba531b4947048aedbe36a434be95"
format_stamp: "Formatted at 2026-06-19 08:59:55 on dist-test-slave-h4g5"
server_key: "788cbb79a7392bbabea1a1276b87245d"
server_key_iv: "6805a176965b6229d6ab87f2781d3710"
server_key_version: "encryptionkey@0"
I20260619 08:59:55.106801 19358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0b12ba531b4947048aedbe36a434be95"
format_stamp: "Formatted at 2026-06-19 08:59:55 on dist-test-slave-h4g5"
server_key: "788cbb79a7392bbabea1a1276b87245d"
server_key_iv: "6805a176965b6229d6ab87f2781d3710"
server_key_version: "encryptionkey@0"
I20260619 08:59:55.108011 19358 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:59:55.108825 19372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:55.109036 19358 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:59:55.109126 19358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-2/wal
uuid: "0b12ba531b4947048aedbe36a434be95"
format_stamp: "Formatted at 2026-06-19 08:59:55 on dist-test-slave-h4g5"
server_key: "788cbb79a7392bbabea1a1276b87245d"
server_key_iv: "6805a176965b6229d6ab87f2781d3710"
server_key_version: "encryptionkey@0"
I20260619 08:59:55.109198 19358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:59:55.149712 19358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:59:55.150018 19358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:59:55.150149 19358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:59:55.150375 19358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:59:55.150686 19358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:59:55.150718 19358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:55.150748 19358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:59:55.150764 19358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:55.156011 19358 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42289
I20260619 08:59:55.156057 19485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42289 every 8 connection(s)
I20260619 08:59:55.156388 19358 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
I20260619 08:59:55.158638 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 19358
I20260619 08:59:55.158727 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-2/wal/instance
I20260619 08:59:55.158888 392 external_mini_cluster.cc:1428] Setting key 52a691538d130190948b8b0d41ad0e77
I20260619 08:59:55.161302 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:55.157023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37481 (local address 127.0.98.62:46109)
0619 08:59:55.157067 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:55.157069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:55.157311 (+ 242us) server_negotiation.cc:407] Connection header received
0619 08:59:55.157804 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:55.157805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:55.157817 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:55.157869 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:55.158521 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:55.159118 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:55.160063 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:55.160282 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:55.160337 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:55.161036 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:55.161038 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:55.161038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:59:55.161042 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:55.161057 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:59:55.161061 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:55.161104 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:55.161198 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 08:59:55.161225 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 08:59:55.161478 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:55.157015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46109 (local address 127.0.98.3:37481)
0619 08:59:55.157191 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:55.157224 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:55.157709 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:55.157938 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:55.157942 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:55.157952 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:59:55.158432 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:55.158437 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:55.159200 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:55.159201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:55.159982 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:55.159984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:55.160354 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:55.160355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:55.160477 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:55.160915 (+ 438us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:59:55.160915 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:59:55.160942 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:59:55.160964 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:59:55.161135 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:59:55.161139 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:59:55.161141 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:59:55.161197 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 08:59:55.161294 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":73,"threads_started":1}
I20260619 08:59:55.161865 19486 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46109
I20260619 08:59:55.161942 19486 heartbeater.cc:464] Registering TS with master...
I20260619 08:59:55.162101 19486 heartbeater.cc:511] Master 127.0.98.62:46109 requested a full tablet report, sending...
I20260619 08:59:55.162420 19036 ts_manager.cc:194] Registered new tserver with Master: 0b12ba531b4947048aedbe36a434be95 (127.0.98.3:42289)
I20260619 08:59:55.162767 19036 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:37481
I20260619 08:59:55.170502 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:59:55.174646 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:55.171865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47662 (local address 127.0.98.62:46109)
0619 08:59:55.171917 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:55.171919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:55.172005 (+ 86us) server_negotiation.cc:407] Connection header received
0619 08:59:55.172068 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:55.172068 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:55.172079 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:55.172141 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:55.172299 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:55.172768 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:55.173488 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:55.173699 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:55.173752 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:55.174373 (+ 621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:55.174374 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:55.174375 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:59:55.174379 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:55.174392 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:59:55.174396 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:55.174470 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:55.174555 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 08:59:55.174566 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 08:59:55.176182 19036 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47662:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:59:55.177693 19036 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:59:55.183485 19496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:55.179638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38277 (local address 127.0.0.1:52564)
0619 08:59:55.179976 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:55.179985 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:55.180033 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:55.180258 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:55.180261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:55.180264 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:59:55.180335 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:55.180338 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:55.181639 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:55.181640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:55.182510 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:55.182513 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:55.183291 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:55.183291 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:55.183342 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:55.183352 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 08:59:55.183397 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 08:59:55.183431 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":28,"thread_start_us":132,"threads_started":1}
I20260619 08:59:55.183521 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:55.179899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52564 (local address 127.0.98.1:38277)
0619 08:59:55.180067 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:55.180071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:55.180110 (+ 39us) server_negotiation.cc:407] Connection header received
0619 08:59:55.180150 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:55.180152 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:55.180167 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:55.180230 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:59:55.180409 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:55.181539 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:55.182600 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:55.183215 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:55.183279 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:55.183362 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:55.183410 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 08:59:55.183443 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":94,"threads_started":1}
I20260619 08:59:55.183959 19494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:55.179638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46529 (local address 127.0.0.1:55588)
0619 08:59:55.180072 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:55.180101 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:55.180142 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:55.180315 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:55.180316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:55.180318 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:59:55.180355 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:55.180356 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:55.181613 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:55.181613 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:55.182880 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:55.182883 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:55.183744 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:55.183745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:55.183801 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:55.183805 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:59:55.183854 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 08:59:55.183893 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":101,"thread_start_us":49,"threads_started":1}
I20260619 08:59:55.183984 19497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:55.179818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55588 (local address 127.0.98.2:46529)
0619 08:59:55.180163 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:55.180168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:55.180177 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:59:55.180226 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:55.180228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:55.180242 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:55.180319 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:59:55.180429 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:55.181537 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:55.182965 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:55.183642 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:55.183712 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:55.183793 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:55.183871 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 08:59:55.183907 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":67,"threads_started":1}
I20260619 08:59:55.184190 19158 tablet_service.cc:1467] Processing CreateTablet for tablet 8d246834d5af458e967ec1d2e1a5c7f6 (DEFAULT_TABLE table=test-table [id=8d0214062198432aae69619bb52f8e76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:59:55.184540 19158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d246834d5af458e967ec1d2e1a5c7f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:59:55.184944 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:55.179638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42289 (local address 127.0.0.1:58600)
0619 08:59:55.179798 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:55.179835 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:55.179899 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:55.180596 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:55.180597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:55.180599 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:59:55.180632 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:55.180634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:55.181571 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:55.181572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:55.182888 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:55.182889 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:55.184725 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:55.184726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:55.184787 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:55.184791 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:59:55.184840 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 08:59:55.184879 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":78,"thread_start_us":92,"threads_started":1}
I20260619 08:59:55.185045 19498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:55.179828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58600 (local address 127.0.98.3:42289)
0619 08:59:55.180437 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:55.180440 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:55.180447 (+ 7us) server_negotiation.cc:407] Connection header received
0619 08:59:55.180485 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:55.180487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:55.180501 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:55.180699 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:59:55.180753 (+ 54us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:55.181494 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:55.183826 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:55.184612 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:55.184698 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:55.184779 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:55.184919 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 08:59:55.184971 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":85,"threads_started":1}
I20260619 08:59:55.185859 19420 tablet_service.cc:1467] Processing CreateTablet for tablet 8d246834d5af458e967ec1d2e1a5c7f6 (DEFAULT_TABLE table=test-table [id=8d0214062198432aae69619bb52f8e76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:59:55.186203 19420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d246834d5af458e967ec1d2e1a5c7f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:59:55.188236 19289 tablet_service.cc:1467] Processing CreateTablet for tablet 8d246834d5af458e967ec1d2e1a5c7f6 (DEFAULT_TABLE table=test-table [id=8d0214062198432aae69619bb52f8e76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:59:55.188357 19500 tablet_bootstrap.cc:492] T 8d246834d5af458e967ec1d2e1a5c7f6 P f3d7e04437e24cf68be5bbea6fe9f7cf: Bootstrap starting.
I20260619 08:59:55.188568 19289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d246834d5af458e967ec1d2e1a5c7f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:59:55.188911 19500 tablet_bootstrap.cc:654] T 8d246834d5af458e967ec1d2e1a5c7f6 P f3d7e04437e24cf68be5bbea6fe9f7cf: Neither blocks nor log segments found. Creating new log.
I20260619 08:59:55.189177 19500 log.cc:824] T 8d246834d5af458e967ec1d2e1a5c7f6 P f3d7e04437e24cf68be5bbea6fe9f7cf: Log is configured to *not* fsync() on all Append() calls
I20260619 08:59:55.189373 19501 tablet_bootstrap.cc:492] T 8d246834d5af458e967ec1d2e1a5c7f6 P 0b12ba531b4947048aedbe36a434be95: Bootstrap starting.
I20260619 08:59:55.189942 19501 tablet_bootstrap.cc:654] T 8d246834d5af458e967ec1d2e1a5c7f6 P 0b12ba531b4947048aedbe36a434be95: Neither blocks nor log segments found. Creating new log.
I20260619 08:59:55.189999 19500 tablet_bootstrap.cc:492] T 8d246834d5af458e967ec1d2e1a5c7f6 P f3d7e04437e24cf68be5bbea6fe9f7cf: No bootstrap required, opened a new log
I20260619 08:59:55.190056 19500 ts_tablet_manager.cc:1397] T 8d246834d5af458e967ec1d2e1a5c7f6 P f3d7e04437e24cf68be5bbea6fe9f7cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:59:55.190237 19501 log.cc:824] T 8d246834d5af458e967ec1d2e1a5c7f6 P 0b12ba531b4947048aedbe36a434be95: Log is configured to *not* fsync() on all Append() calls
I20260619 08:59:55.190940 19501 tablet_bootstrap.cc:492] T 8d246834d5af458e967ec1d2e1a5c7f6 P 0b12ba531b4947048aedbe36a434be95: No bootstrap required, opened a new log
I20260619 08:59:55.191005 19501 ts_tablet_manager.cc:1397] T 8d246834d5af458e967ec1d2e1a5c7f6 P 0b12ba531b4947048aedbe36a434be95: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:59:55.191519 19504 tablet_bootstrap.cc:492] T 8d246834d5af458e967ec1d2e1a5c7f6 P 7db7750599754a1381a7e4469b99f684: Bootstrap starting.
I20260619 08:59:55.192023 19504 tablet_bootstrap.cc:654] T 8d246834d5af458e967ec1d2e1a5c7f6 P 7db7750599754a1381a7e4469b99f684: Neither blocks nor log segments found. Creating new log.
I20260619 08:59:55.192315 19504 log.cc:824] T 8d246834d5af458e967ec1d2e1a5c7f6 P 7db7750599754a1381a7e4469b99f684: Log is configured to *not* fsync() on all Append() calls
I20260619 08:59:55.192584 19500 raft_consensus.cc:348] T 8d246834d5af458e967ec1d2e1a5c7f6 P f3d7e04437e24cf68be5bbea6fe9f7cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db7750599754a1381a7e4469b99f684" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46529 } } peers { permanent_uuid: "0b12ba531b4947048aedbe36a434be95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42289 } } peers { permanent_uuid: "f3d7e04437e24cf68be5bbea6fe9f7cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38277 } }
I20260619 08:59:55.192760 19500 raft_consensus.cc:374] T 8d246834d5af458e967ec1d2e1a5c7f6 P f3d7e04437e24cf68be5bbea6fe9f7cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:59:55.192802 19500 raft_consensus.cc:729] T 8d246834d5af458e967ec1d2e1a5c7f6 P f3d7e04437e24cf68be5bbea6fe9f7cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3d7e04437e24cf68be5bbea6fe9f7cf, State: Initialized, Role: FOLLOWER
I20260619 08:59:55.192947 19504 tablet_bootstrap.cc:492] T 8d246834d5af458e967ec1d2e1a5c7f6 P 7db7750599754a1381a7e4469b99f684: No bootstrap required, opened a new log
I20260619 08:59:55.192948 19500 consensus_queue.cc:260] T 8d246834d5af458e967ec1d2e1a5c7f6 P f3d7e04437e24cf68be5bbea6fe9f7cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db7750599754a1381a7e4469b99f684" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46529 } } peers { permanent_uuid: "0b12ba531b4947048aedbe36a434be95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42289 } } peers { permanent_uuid: "f3d7e04437e24cf68be5bbea6fe9f7cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38277 } }
I20260619 08:59:55.193005 19504 ts_tablet_manager.cc:1397] T 8d246834d5af458e967ec1d2e1a5c7f6 P 7db7750599754a1381a7e4469b99f684: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 08:59:55.193212 19500 ts_tablet_manager.cc:1428] T 8d246834d5af458e967ec1d2e1a5c7f6 P f3d7e04437e24cf68be5bbea6fe9f7cf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:59:55.193359 19224 heartbeater.cc:503] Master 127.0.98.62:46109 was elected leader, sending a full tablet report...
I20260619 08:59:55.193499 19501 raft_consensus.cc:348] T 8d246834d5af458e967ec1d2e1a5c7f6 P 0b12ba531b4947048aedbe36a434be95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db7750599754a1381a7e4469b99f684" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46529 } } peers { permanent_uuid: "0b12ba531b4947048aedbe36a434be95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42289 } } peers { permanent_uuid: "f3d7e04437e24cf68be5bbea6fe9f7cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38277 } }
I20260619 08:59:55.193643 19501 raft_consensus.cc:374] T 8d246834d5af458e967ec1d2e1a5c7f6 P 0b12ba531b4947048aedbe36a434be95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:59:55.193688 19501 raft_consensus.cc:729] T 8d246834d5af458e967ec1d2e1a5c7f6 P 0b12ba531b4947048aedbe36a434be95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b12ba531b4947048aedbe36a434be95, State: Initialized, Role: FOLLOWER
I20260619 08:59:55.193799 19501 consensus_queue.cc:260] T 8d246834d5af458e967ec1d2e1a5c7f6 P 0b12ba531b4947048aedbe36a434be95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db7750599754a1381a7e4469b99f684" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46529 } } peers { permanent_uuid: "0b12ba531b4947048aedbe36a434be95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42289 } } peers { permanent_uuid: "f3d7e04437e24cf68be5bbea6fe9f7cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38277 } }
I20260619 08:59:55.194082 19501 ts_tablet_manager.cc:1428] T 8d246834d5af458e967ec1d2e1a5c7f6 P 0b12ba531b4947048aedbe36a434be95: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:59:55.194154 19486 heartbeater.cc:503] Master 127.0.98.62:46109 was elected leader, sending a full tablet report...
I20260619 08:59:55.195447 19504 raft_consensus.cc:348] T 8d246834d5af458e967ec1d2e1a5c7f6 P 7db7750599754a1381a7e4469b99f684 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db7750599754a1381a7e4469b99f684" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46529 } } peers { permanent_uuid: "0b12ba531b4947048aedbe36a434be95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42289 } } peers { permanent_uuid: "f3d7e04437e24cf68be5bbea6fe9f7cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38277 } }
I20260619 08:59:55.195600 19504 raft_consensus.cc:374] T 8d246834d5af458e967ec1d2e1a5c7f6 P 7db7750599754a1381a7e4469b99f684 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:59:55.195640 19504 raft_consensus.cc:729] T 8d246834d5af458e967ec1d2e1a5c7f6 P 7db7750599754a1381a7e4469b99f684 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7db7750599754a1381a7e4469b99f684, State: Initialized, Role: FOLLOWER
I20260619 08:59:55.195741 19504 consensus_queue.cc:260] T 8d246834d5af458e967ec1d2e1a5c7f6 P 7db7750599754a1381a7e4469b99f684 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db7750599754a1381a7e4469b99f684" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46529 } } peers { permanent_uuid: "0b12ba531b4947048aedbe36a434be95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42289 } } peers { permanent_uuid: "f3d7e04437e24cf68be5bbea6fe9f7cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38277 } }
I20260619 08:59:55.195945 19504 ts_tablet_manager.cc:1428] T 8d246834d5af458e967ec1d2e1a5c7f6 P 7db7750599754a1381a7e4469b99f684: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 08:59:55.195990 19355 heartbeater.cc:503] Master 127.0.98.62:46109 was elected leader, sending a full tablet report...
W20260619 08:59:55.258764 19356 tablet.cc:2369] T 8d246834d5af458e967ec1d2e1a5c7f6 P 7db7750599754a1381a7e4469b99f684: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:59:55.269305 19507 raft_consensus.cc:482] T 8d246834d5af458e967ec1d2e1a5c7f6 P 0b12ba531b4947048aedbe36a434be95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:59:55.269419 19507 raft_consensus.cc:504] T 8d246834d5af458e967ec1d2e1a5c7f6 P 0b12ba531b4947048aedbe36a434be95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db7750599754a1381a7e4469b99f684" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46529 } } peers { permanent_uuid: "0b12ba531b4947048aedbe36a434be95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42289 } } peers { permanent_uuid: "f3d7e04437e24cf68be5bbea6fe9f7cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38277 } }
I20260619 08:59:55.269778 19507 leader_election.cc:288] T 8d246834d5af458e967ec1d2e1a5c7f6 P 0b12ba531b4947048aedbe36a434be95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7db7750599754a1381a7e4469b99f684 (127.0.98.2:46529), f3d7e04437e24cf68be5bbea6fe9f7cf (127.0.98.1:38277)
I20260619 08:59:55.273111 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:55.270066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40787 (local address 127.0.98.1:38277)
0619 08:59:55.270128 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:55.270130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:55.270145 (+ 15us) server_negotiation.cc:407] Connection header received
0619 08:59:55.270195 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:55.270196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:55.270207 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:55.270258 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:59:55.270442 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:55.271233 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:55.272390 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:55.272786 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:55.272846 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:55.272858 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:55.273009 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 08:59:55.273050 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 08:59:55.273106 19497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:55.270109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51551 (local address 127.0.98.2:46529)
0619 08:59:55.270155 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:55.270157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:55.270295 (+ 138us) server_negotiation.cc:407] Connection header received
0619 08:59:55.270342 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:55.270343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:55.270356 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:55.270406 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:59:55.270530 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:55.271266 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:55.272418 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:55.272781 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:55.272852 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:55.272863 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:55.272996 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 08:59:55.273033 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 08:59:55.273108 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:55.269996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38277 (local address 127.0.98.3:40787)
0619 08:59:55.270048 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:55.270061 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:55.270124 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:55.270285 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:55.270286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:55.270288 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:59:55.270376 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:55.270378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:55.271346 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:55.271348 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:55.272293 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:55.272296 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:55.272866 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:55.272867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:55.272933 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:55.272936 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:59:55.272986 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 08:59:55.273027 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":32}
I20260619 08:59:55.273120 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:55.269996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46529 (local address 127.0.98.3:51551)
0619 08:59:55.270230 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:55.270239 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:55.270285 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:55.270414 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:55.270415 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:55.270416 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:59:55.270471 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:55.270472 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:55.271342 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:55.271343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:55.272335 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:55.272338 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:55.272862 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:55.272863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:55.272934 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:55.272937 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:59:55.272980 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 08:59:55.273018 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":108,"thread_start_us":96,"threads_started":1}
I20260619 08:59:55.273540 19178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d246834d5af458e967ec1d2e1a5c7f6" candidate_uuid: "0b12ba531b4947048aedbe36a434be95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3d7e04437e24cf68be5bbea6fe9f7cf" is_pre_election: true
I20260619 08:59:55.273734 19178 raft_consensus.cc:2454] T 8d246834d5af458e967ec1d2e1a5c7f6 P f3d7e04437e24cf68be5bbea6fe9f7cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b12ba531b4947048aedbe36a434be95 in term 0.
I20260619 08:59:55.273689 19309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d246834d5af458e967ec1d2e1a5c7f6" candidate_uuid: "0b12ba531b4947048aedbe36a434be95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7db7750599754a1381a7e4469b99f684" is_pre_election: true
I20260619 08:59:55.273833 19309 raft_consensus.cc:2454] T 8d246834d5af458e967ec1d2e1a5c7f6 P 7db7750599754a1381a7e4469b99f684 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b12ba531b4947048aedbe36a434be95 in term 0.
I20260619 08:59:55.273972 19376 leader_election.cc:302] T 8d246834d5af458e967ec1d2e1a5c7f6 P 0b12ba531b4947048aedbe36a434be95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b12ba531b4947048aedbe36a434be95, f3d7e04437e24cf68be5bbea6fe9f7cf; no voters:
I20260619 08:59:55.274116 19507 raft_consensus.cc:2781] T 8d246834d5af458e967ec1d2e1a5c7f6 P 0b12ba531b4947048aedbe36a434be95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:59:55.274176 19507 raft_consensus.cc:482] T 8d246834d5af458e967ec1d2e1a5c7f6 P 0b12ba531b4947048aedbe36a434be95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:59:55.274192 19507 raft_consensus.cc:3037] T 8d246834d5af458e967ec1d2e1a5c7f6 P 0b12ba531b4947048aedbe36a434be95 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:59:55.274870 19507 raft_consensus.cc:504] T 8d246834d5af458e967ec1d2e1a5c7f6 P 0b12ba531b4947048aedbe36a434be95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db7750599754a1381a7e4469b99f684" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46529 } } peers { permanent_uuid: "0b12ba531b4947048aedbe36a434be95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42289 } } peers { permanent_uuid: "f3d7e04437e24cf68be5bbea6fe9f7cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38277 } }
I20260619 08:59:55.274976 19507 leader_election.cc:288] T 8d246834d5af458e967ec1d2e1a5c7f6 P 0b12ba531b4947048aedbe36a434be95 [CANDIDATE]: Term 1 election: Requested vote from peers 7db7750599754a1381a7e4469b99f684 (127.0.98.2:46529), f3d7e04437e24cf68be5bbea6fe9f7cf (127.0.98.1:38277)
I20260619 08:59:55.275166 19178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d246834d5af458e967ec1d2e1a5c7f6" candidate_uuid: "0b12ba531b4947048aedbe36a434be95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3d7e04437e24cf68be5bbea6fe9f7cf"
I20260619 08:59:55.275192 19309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d246834d5af458e967ec1d2e1a5c7f6" candidate_uuid: "0b12ba531b4947048aedbe36a434be95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7db7750599754a1381a7e4469b99f684"
I20260619 08:59:55.275250 19178 raft_consensus.cc:3037] T 8d246834d5af458e967ec1d2e1a5c7f6 P f3d7e04437e24cf68be5bbea6fe9f7cf [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:59:55.275267 19309 raft_consensus.cc:3037] T 8d246834d5af458e967ec1d2e1a5c7f6 P 7db7750599754a1381a7e4469b99f684 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:59:55.275939 19178 raft_consensus.cc:2454] T 8d246834d5af458e967ec1d2e1a5c7f6 P f3d7e04437e24cf68be5bbea6fe9f7cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b12ba531b4947048aedbe36a434be95 in term 1.
I20260619 08:59:55.276121 19376 leader_election.cc:302] T 8d246834d5af458e967ec1d2e1a5c7f6 P 0b12ba531b4947048aedbe36a434be95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b12ba531b4947048aedbe36a434be95, f3d7e04437e24cf68be5bbea6fe9f7cf; no voters:
I20260619 08:59:55.276141 19309 raft_consensus.cc:2454] T 8d246834d5af458e967ec1d2e1a5c7f6 P 7db7750599754a1381a7e4469b99f684 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b12ba531b4947048aedbe36a434be95 in term 1.
I20260619 08:59:55.276250 19507 raft_consensus.cc:2781] T 8d246834d5af458e967ec1d2e1a5c7f6 P 0b12ba531b4947048aedbe36a434be95 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:59:55.276397 19507 raft_consensus.cc:686] T 8d246834d5af458e967ec1d2e1a5c7f6 P 0b12ba531b4947048aedbe36a434be95 [term 1 LEADER]: Becoming Leader. State: Replica: 0b12ba531b4947048aedbe36a434be95, State: Running, Role: LEADER
I20260619 08:59:55.276481 19507 consensus_queue.cc:237] T 8d246834d5af458e967ec1d2e1a5c7f6 P 0b12ba531b4947048aedbe36a434be95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db7750599754a1381a7e4469b99f684" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46529 } } peers { permanent_uuid: "0b12ba531b4947048aedbe36a434be95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42289 } } peers { permanent_uuid: "f3d7e04437e24cf68be5bbea6fe9f7cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38277 } }
I20260619 08:59:55.277346 19036 catalog_manager.cc:5495] T 8d246834d5af458e967ec1d2e1a5c7f6 P 0b12ba531b4947048aedbe36a434be95 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b12ba531b4947048aedbe36a434be95 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0b12ba531b4947048aedbe36a434be95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db7750599754a1381a7e4469b99f684" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b12ba531b4947048aedbe36a434be95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3d7e04437e24cf68be5bbea6fe9f7cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38277 } health_report { overall_health: UNKNOWN } } }
I20260619 08:59:55.298144 19498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:55.295846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58616 (local address 127.0.98.3:42289)
0619 08:59:55.295896 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:55.295903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:55.295913 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:59:55.295997 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:55.295997 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:55.296019 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:55.296075 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:59:55.296238 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:55.296711 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:55.297441 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:55.297623 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:55.297678 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:55.297819 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:59:55.297907 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:59:55.297990 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:55.298036 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 08:59:55.298043 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 08:59:55.299897 19178 raft_consensus.cc:1261] T 8d246834d5af458e967ec1d2e1a5c7f6 P f3d7e04437e24cf68be5bbea6fe9f7cf [term 1 FOLLOWER]: Refusing update from remote peer 0b12ba531b4947048aedbe36a434be95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:59:55.299899 19309 raft_consensus.cc:1261] T 8d246834d5af458e967ec1d2e1a5c7f6 P 7db7750599754a1381a7e4469b99f684 [term 1 FOLLOWER]: Refusing update from remote peer 0b12ba531b4947048aedbe36a434be95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:59:55.300276 19510 consensus_queue.cc:1035] T 8d246834d5af458e967ec1d2e1a5c7f6 P 0b12ba531b4947048aedbe36a434be95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7db7750599754a1381a7e4469b99f684" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:59:55.300407 19507 consensus_queue.cc:1035] T 8d246834d5af458e967ec1d2e1a5c7f6 P 0b12ba531b4947048aedbe36a434be95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3d7e04437e24cf68be5bbea6fe9f7cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:59:55.302649 19513 mvcc.cc:205] Tried to move back new op lower bound from 7298496902345523200 to 7298496902252949504. Current Snapshot: MvccSnapshot[applied={T|T < 7298496902345523200}]
I20260619 08:59:55.302773 19514 mvcc.cc:205] Tried to move back new op lower bound from 7298496902345523200 to 7298496902252949504. Current Snapshot: MvccSnapshot[applied={T|T < 7298496902345523200}]
I20260619 08:59:55.303112 19515 mvcc.cc:205] Tried to move back new op lower bound from 7298496902345523200 to 7298496902252949504. Current Snapshot: MvccSnapshot[applied={T|T < 7298496902345523200}]
I20260619 08:59:55.304674 19035 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:59:55.304752 19035 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:59:55.305419 19035 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 721abaf638d447f59e019d36ef073187: Sending DeleteTablet for 3 replicas of tablet 8d246834d5af458e967ec1d2e1a5c7f6
I20260619 08:59:55.305794 19420 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d246834d5af458e967ec1d2e1a5c7f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:59:55 UTC) from {username='slave'} at 127.0.0.1:58600
I20260619 08:59:55.305786 19158 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d246834d5af458e967ec1d2e1a5c7f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:59:55 UTC) from {username='slave'} at 127.0.0.1:52564
I20260619 08:59:55.306011 19289 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d246834d5af458e967ec1d2e1a5c7f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:59:55 UTC) from {username='slave'} at 127.0.0.1:55588
I20260619 08:59:55.306030 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 19096
I20260619 08:59:55.307289 19524 tablet_replica.cc:330] T 8d246834d5af458e967ec1d2e1a5c7f6 P 0b12ba531b4947048aedbe36a434be95: stopping tablet replica
I20260619 08:59:55.307370 19524 raft_consensus.cc:2229] T 8d246834d5af458e967ec1d2e1a5c7f6 P 0b12ba531b4947048aedbe36a434be95 [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:59:55.307448 19524 raft_consensus.cc:2258] T 8d246834d5af458e967ec1d2e1a5c7f6 P 0b12ba531b4947048aedbe36a434be95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:59:55.307884 19524 ts_tablet_manager.cc:1905] T 8d246834d5af458e967ec1d2e1a5c7f6 P 0b12ba531b4947048aedbe36a434be95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:59:55.309499 19524 ts_tablet_manager.cc:1918] T 8d246834d5af458e967ec1d2e1a5c7f6 P 0b12ba531b4947048aedbe36a434be95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:59:55.309568 19524 log.cc:1196] T 8d246834d5af458e967ec1d2e1a5c7f6 P 0b12ba531b4947048aedbe36a434be95: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859361540516-392-0/minicluster-data/ts-2/wal/wals/8d246834d5af458e967ec1d2e1a5c7f6
I20260619 08:59:55.309801 19524 ts_tablet_manager.cc:1939] T 8d246834d5af458e967ec1d2e1a5c7f6 P 0b12ba531b4947048aedbe36a434be95: Deleting consensus metadata
I20260619 08:59:55.310171 19022 catalog_manager.cc:4841] TS 0b12ba531b4947048aedbe36a434be95 (127.0.98.3:42289): tablet 8d246834d5af458e967ec1d2e1a5c7f6 (table test-table [id=8d0214062198432aae69619bb52f8e76]) successfully deleted
W20260619 08:59:55.311832 19023 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38277 (error 108)
I20260619 08:59:55.312188 19525 tablet_replica.cc:330] T 8d246834d5af458e967ec1d2e1a5c7f6 P 7db7750599754a1381a7e4469b99f684: stopping tablet replica
I20260619 08:59:55.312389 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 19227
I20260619 08:59:55.319751 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:55.319587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46529 (local address 127.0.0.1:55592)
0619 08:59:55.319647 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:55.319702 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 08:59:55.319893 19021 catalog_manager.cc:4584] TS 7db7750599754a1381a7e4469b99f684 (127.0.98.2:46529): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8d246834d5af458e967ec1d2e1a5c7f6: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46529: connect: Connection refused (error 111)
I20260619 08:59:55.320714 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:55.320599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38277 (local address 127.0.0.1:52576)
0619 08:59:55.320648 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:55.320679 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 08:59:55.320716 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 19358
I20260619 08:59:55.325162 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 19005
2026-06-19T08:59:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (728 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:59:55 dist-test-slave-h4g5 krb5kdc[19529](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:59:55 dist-test-slave-h4g5 krb5kdc[19529](info): set up 2 sockets
Jun 19 08:59:55 dist-test-slave-h4g5 krb5kdc[19529](info): commencing operation
krb5kdc: starting...
W20260619 08:59:56.271690 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 08:59:56 dist-test-slave-h4g5 krb5kdc[19529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:59:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:59:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 08:59:56.384654 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39751
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39751
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:59:56.461817 19542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:59:56.462033 19542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:59:56.462064 19542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:59:56.463188 19542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:59:56.463229 19542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:59:56.463248 19542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:59:56.463264 19542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:59:56.463279 19542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:59:56.464727 19542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:59:56.464962 19542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:59:56.465157 19542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:59:56.469766 19547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:56.469766 19550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:56.469959 19542 server_base.cc:1034] running on GCE node
W20260619 08:59:56.469790 19548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:56.470260 19542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:59:56.470518 19542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:59:56.471655 19542 hybrid_clock.cc:648] HybridClock initialized: now 1781859596471642 us; error 30 us; skew 500 ppm
Jun 19 08:59:56 dist-test-slave-h4g5 krb5kdc[19529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:59:56.474038 19542 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 08:59:56.474442 19542 webserver.cc:460] Webserver started at http://127.0.98.62:34873/ using document root <none> and password file <none>
I20260619 08:59:56.474625 19542 fs_manager.cc:359] Metadata directory not provided
I20260619 08:59:56.474665 19542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:59:56.474757 19542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:59:56.475742 19542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "f4fb1772c6ba4c26925d8204a64a926a"
format_stamp: "Formatted at 2026-06-19 08:59:56 on dist-test-slave-h4g5"
server_key: "f79f9c93c94a541c29883bb2eb029eac"
server_key_iv: "d9982ff26829305e13f823318d429f7d"
server_key_version: "encryptionkey@0"
I20260619 08:59:56.476032 19542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "f4fb1772c6ba4c26925d8204a64a926a"
format_stamp: "Formatted at 2026-06-19 08:59:56 on dist-test-slave-h4g5"
server_key: "f79f9c93c94a541c29883bb2eb029eac"
server_key_iv: "d9982ff26829305e13f823318d429f7d"
server_key_version: "encryptionkey@0"
I20260619 08:59:56.477247 19542 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:59:56.477837 19557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:56.477994 19542 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 08:59:56.478077 19542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "f4fb1772c6ba4c26925d8204a64a926a"
format_stamp: "Formatted at 2026-06-19 08:59:56 on dist-test-slave-h4g5"
server_key: "f79f9c93c94a541c29883bb2eb029eac"
server_key_iv: "d9982ff26829305e13f823318d429f7d"
server_key_version: "encryptionkey@0"
I20260619 08:59:56.478135 19542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:59:56.516247 19542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:59:56.517968 19542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:59:56.518072 19542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:59:56.521582 19542 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39751
I20260619 08:59:56.521656 19609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39751 every 8 connection(s)
I20260619 08:59:56.521951 19542 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:59:56.522753 19610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:59:56.524950 19610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4fb1772c6ba4c26925d8204a64a926a: Bootstrap starting.
I20260619 08:59:56.525470 19610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4fb1772c6ba4c26925d8204a64a926a: Neither blocks nor log segments found. Creating new log.
I20260619 08:59:56.525713 19610 log.cc:824] T 00000000000000000000000000000000 P f4fb1772c6ba4c26925d8204a64a926a: Log is configured to *not* fsync() on all Append() calls
I20260619 08:59:56.526328 19610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4fb1772c6ba4c26925d8204a64a926a: No bootstrap required, opened a new log
I20260619 08:59:56.527876 19610 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4fb1772c6ba4c26925d8204a64a926a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4fb1772c6ba4c26925d8204a64a926a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39751 } }
I20260619 08:59:56.528012 19610 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4fb1772c6ba4c26925d8204a64a926a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:59:56.528043 19610 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4fb1772c6ba4c26925d8204a64a926a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4fb1772c6ba4c26925d8204a64a926a, State: Initialized, Role: FOLLOWER
I20260619 08:59:56.528183 19610 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4fb1772c6ba4c26925d8204a64a926a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4fb1772c6ba4c26925d8204a64a926a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39751 } }
I20260619 08:59:56.528247 19610 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4fb1772c6ba4c26925d8204a64a926a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:59:56.528275 19610 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4fb1772c6ba4c26925d8204a64a926a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:59:56.528321 19610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4fb1772c6ba4c26925d8204a64a926a [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:59:56.528796 19610 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4fb1772c6ba4c26925d8204a64a926a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4fb1772c6ba4c26925d8204a64a926a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39751 } }
I20260619 08:59:56.528891 19610 leader_election.cc:302] T 00000000000000000000000000000000 P f4fb1772c6ba4c26925d8204a64a926a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4fb1772c6ba4c26925d8204a64a926a; no voters:
I20260619 08:59:56.529038 19610 leader_election.cc:288] T 00000000000000000000000000000000 P f4fb1772c6ba4c26925d8204a64a926a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:59:56.529098 19613 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4fb1772c6ba4c26925d8204a64a926a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:59:56.529256 19610 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4fb1772c6ba4c26925d8204a64a926a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:59:56.529273 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 19542
I20260619 08:59:56.529291 19613 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4fb1772c6ba4c26925d8204a64a926a [term 1 LEADER]: Becoming Leader. State: Replica: f4fb1772c6ba4c26925d8204a64a926a, State: Running, Role: LEADER
I20260619 08:59:56.529374 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 08:59:56.529387 19613 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4fb1772c6ba4c26925d8204a64a926a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4fb1772c6ba4c26925d8204a64a926a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39751 } }
I20260619 08:59:56.529603 392 external_mini_cluster.cc:1428] Setting key ddb5b6b9e3607e3603a21198c128b486
I20260619 08:59:56.529850 19615 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4fb1772c6ba4c26925d8204a64a926a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4fb1772c6ba4c26925d8204a64a926a. Latest consensus state: current_term: 1 leader_uuid: "f4fb1772c6ba4c26925d8204a64a926a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4fb1772c6ba4c26925d8204a64a926a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39751 } } }
I20260619 08:59:56.529851 19614 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4fb1772c6ba4c26925d8204a64a926a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4fb1772c6ba4c26925d8204a64a926a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4fb1772c6ba4c26925d8204a64a926a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39751 } } }
I20260619 08:59:56.529965 19615 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4fb1772c6ba4c26925d8204a64a926a [sys.catalog]: This master's current role is: LEADER
I20260619 08:59:56.529969 19614 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4fb1772c6ba4c26925d8204a64a926a [sys.catalog]: This master's current role is: LEADER
I20260619 08:59:56.530187 19623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:59:56.530684 19623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:59:56.533416 19623 catalog_manager.cc:1347] Generated new cluster ID: 5517cd2f56d6416b885ff29c3a16f18e
I20260619 08:59:56.533486 19623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 08:59:56 dist-test-slave-h4g5 krb5kdc[19529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:59:56.539922 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:56.530277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55690 (local address 127.0.98.62:39751)
0619 08:59:56.530524 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:56.530533 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:56.530712 (+ 179us) server_negotiation.cc:407] Connection header received
0619 08:59:56.530913 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:56.530921 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:56.530981 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:56.531175 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:56.531839 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:56.533208 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:56.534044 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:56.534472 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:56.534564 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:56.536712 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:56.536716 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:56.536716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:59:56.536727 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:56.538598 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:56.538901 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:56.538904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:56.538904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:56.538993 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:56.539160 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:56.539161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:56.539162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:56.539385 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:56.539529 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:56.539602 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 08:59:56.539729 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":65,"threads_started":1}
I20260619 08:59:56.543816 19623 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:59:56.544541 19623 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:59:56.554184 19623 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4fb1772c6ba4c26925d8204a64a926a: Generated new TSK 0
I20260619 08:59:56.554410 19623 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 08:59:56.585088 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39751
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:59:56.671813 19636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:59:56.671996 19636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:59:56.672019 19636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:59:56.673152 19636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:59:56.673192 19636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:59:56.673220 19636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 08:59:56.674753 19636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:59:56.674937 19636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:59:56.675109 19636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:59:56.679711 19641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:56.679718 19642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:56.679872 19644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:56.679967 19636 server_base.cc:1034] running on GCE node
I20260619 08:59:56.680179 19636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:59:56.680387 19636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:59:56.681535 19636 hybrid_clock.cc:648] HybridClock initialized: now 1781859596681516 us; error 32 us; skew 500 ppm
Jun 19 08:59:56 dist-test-slave-h4g5 krb5kdc[19529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:59:56.683911 19636 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 08:59:56.684374 19636 webserver.cc:460] Webserver started at http://127.0.98.1:44653/ using document root <none> and password file <none>
I20260619 08:59:56.684579 19636 fs_manager.cc:359] Metadata directory not provided
I20260619 08:59:56.684625 19636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:59:56.684747 19636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:59:56.685750 19636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-0/data/instance:
uuid: "4835edbbf16f44b1b76dd3cfeb7d1b54"
format_stamp: "Formatted at 2026-06-19 08:59:56 on dist-test-slave-h4g5"
server_key: "ab3cf85eccc9d7e8c27597d968dde31c"
server_key_iv: "e4147afb0221f0ce390df66a0ff05da7"
server_key_version: "encryptionkey@0"
I20260619 08:59:56.686069 19636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4835edbbf16f44b1b76dd3cfeb7d1b54"
format_stamp: "Formatted at 2026-06-19 08:59:56 on dist-test-slave-h4g5"
server_key: "ab3cf85eccc9d7e8c27597d968dde31c"
server_key_iv: "e4147afb0221f0ce390df66a0ff05da7"
server_key_version: "encryptionkey@0"
I20260619 08:59:56.687224 19636 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:59:56.687875 19651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:56.688016 19636 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:59:56.688117 19636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-0/wal
uuid: "4835edbbf16f44b1b76dd3cfeb7d1b54"
format_stamp: "Formatted at 2026-06-19 08:59:56 on dist-test-slave-h4g5"
server_key: "ab3cf85eccc9d7e8c27597d968dde31c"
server_key_iv: "e4147afb0221f0ce390df66a0ff05da7"
server_key_version: "encryptionkey@0"
I20260619 08:59:56.688176 19636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:59:56.702318 19636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:59:56.704396 19636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:59:56.704532 19636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:59:56.704741 19636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:59:56.705027 19636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:59:56.705060 19636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:56.705090 19636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:59:56.705106 19636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:56.710582 19636 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42901
I20260619 08:59:56.710639 19764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42901 every 8 connection(s)
I20260619 08:59:56.710916 19636 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 08:59:56 dist-test-slave-h4g5 krb5kdc[19529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:59:56.719329 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:56.711590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40493 (local address 127.0.98.62:39751)
0619 08:59:56.711648 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:56.711650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:56.711827 (+ 177us) server_negotiation.cc:407] Connection header received
0619 08:59:56.712827 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:56.712829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:56.712842 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:56.712889 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:56.713486 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:56.713988 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:56.714954 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:56.715155 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:56.715208 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:56.717222 (+ 2014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:56.717223 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:56.717224 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:59:56.717229 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:56.718308 (+ 1079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:56.718646 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:56.718647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:56.718647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:56.718689 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:56.718872 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:56.718872 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:56.718873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:56.718964 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:56.719007 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:56.719207 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 08:59:56.719267 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 08:59:56.719516 19767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:56.711547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39751 (local address 127.0.98.1:40493)
0619 08:59:56.711713 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:56.711751 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:56.712729 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:56.712961 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:56.712965 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:56.713203 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:59:56.713390 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:56.713394 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:56.714065 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:56.714066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:56.714875 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:56.714877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:56.715220 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:56.715221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:56.715338 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:56.715759 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:59:56.715767 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:59:56.717141 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:59:56.718389 (+ 1248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:59:56.718395 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:59:56.718397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:59:56.718595 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:59:56.718732 (+ 137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:59:56.718732 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:59:56.718732 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:59:56.718828 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:59:56.719005 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:59:56.719008 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:59:56.719116 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 08:59:56.719190 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 08:59:56.719307 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":85,"threads_started":1}
I20260619 08:59:56.719987 19765 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39751
I20260619 08:59:56.720072 19765 heartbeater.cc:464] Registering TS with master...
I20260619 08:59:56.720162 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 19636
I20260619 08:59:56.720273 19765 heartbeater.cc:511] Master 127.0.98.62:39751 requested a full tablet report, sending...
I20260619 08:59:56.720259 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-0/wal/instance
I20260619 08:59:56.720527 392 external_mini_cluster.cc:1428] Setting key 8116d274e6e3fdc2e85fbdf342f7c936
I20260619 08:59:56.720820 19574 ts_manager.cc:194] Registered new tserver with Master: 4835edbbf16f44b1b76dd3cfeb7d1b54 (127.0.98.1:42901)
I20260619 08:59:56.721539 19574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40493
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 08:59:56.744338 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39751
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:59:56.823276 19770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:59:56.823480 19770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:59:56.823503 19770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:59:56.824661 19770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:59:56.824702 19770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:59:56.824729 19770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 08:59:56.826192 19770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:59:56.826416 19770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:59:56.826612 19770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:59:56.831385 19776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:56.831390 19775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:56.831406 19778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:56.831676 19770 server_base.cc:1034] running on GCE node
I20260619 08:59:56.831878 19770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:59:56.832113 19770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:59:56.833261 19770 hybrid_clock.cc:648] HybridClock initialized: now 1781859596833247 us; error 26 us; skew 500 ppm
Jun 19 08:59:56 dist-test-slave-h4g5 krb5kdc[19529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:59:56.835583 19770 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 08:59:56.836056 19770 webserver.cc:460] Webserver started at http://127.0.98.2:33753/ using document root <none> and password file <none>
I20260619 08:59:56.836282 19770 fs_manager.cc:359] Metadata directory not provided
I20260619 08:59:56.836328 19770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:59:56.836416 19770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:59:56.837361 19770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-1/data/instance:
uuid: "ef22e9e54f654292bfdbc6dd0b6dc09d"
format_stamp: "Formatted at 2026-06-19 08:59:56 on dist-test-slave-h4g5"
server_key: "d309023baf6f3c0ef889ea52e61a2e01"
server_key_iv: "46f67e1abb6d4bd6e1997e2dae91b7dd"
server_key_version: "encryptionkey@0"
I20260619 08:59:56.837641 19770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ef22e9e54f654292bfdbc6dd0b6dc09d"
format_stamp: "Formatted at 2026-06-19 08:59:56 on dist-test-slave-h4g5"
server_key: "d309023baf6f3c0ef889ea52e61a2e01"
server_key_iv: "46f67e1abb6d4bd6e1997e2dae91b7dd"
server_key_version: "encryptionkey@0"
I20260619 08:59:56.838748 19770 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 08:59:56.839363 19785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:56.839496 19770 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:59:56.839568 19770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-1/wal
uuid: "ef22e9e54f654292bfdbc6dd0b6dc09d"
format_stamp: "Formatted at 2026-06-19 08:59:56 on dist-test-slave-h4g5"
server_key: "d309023baf6f3c0ef889ea52e61a2e01"
server_key_iv: "46f67e1abb6d4bd6e1997e2dae91b7dd"
server_key_version: "encryptionkey@0"
I20260619 08:59:56.839633 19770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:59:56.851027 19770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:59:56.852949 19770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:59:56.853070 19770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:59:56.853274 19770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:59:56.853579 19770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:59:56.853611 19770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:56.853633 19770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:59:56.853655 19770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:56.858949 19770 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46633
I20260619 08:59:56.859019 19898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46633 every 8 connection(s)
I20260619 08:59:56.859273 19770 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 08:59:56 dist-test-slave-h4g5 krb5kdc[19529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:59:56.867615 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:56.859946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33313 (local address 127.0.98.62:39751)
0619 08:59:56.860005 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:56.860007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:56.860171 (+ 164us) server_negotiation.cc:407] Connection header received
0619 08:59:56.861070 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:56.861071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:56.861080 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:56.861134 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:56.861686 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:56.862194 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:56.863136 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:56.863319 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:56.863364 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:56.865275 (+ 1911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:56.865276 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:56.865277 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:59:56.865283 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:56.866507 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:56.866909 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:56.866911 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:56.866911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:56.866972 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:56.867141 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:56.867142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:56.867142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:56.867238 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:56.867282 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:56.867487 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 08:59:56.867549 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 08:59:56.867790 19901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:56.859899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39751 (local address 127.0.98.2:33313)
0619 08:59:56.860042 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:56.860102 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:56.860989 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:56.861190 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:56.861193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:56.861427 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:59:56.861610 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:56.861614 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:56.862271 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:56.862272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:56.863064 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:56.863066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:56.863374 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:56.863375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:56.863465 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:56.863886 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:59:56.863894 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:59:56.865190 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:59:56.866608 (+ 1418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:59:56.866613 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:59:56.866615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:59:56.866838 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:59:56.867016 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:59:56.867016 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:59:56.867017 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:59:56.867099 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:59:56.867281 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:59:56.867283 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:59:56.867395 (+ 112us) client_negotiation.cc:769] Sending connection context
0619 08:59:56.867470 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 08:59:56.867588 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
I20260619 08:59:56.868284 19899 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39751
I20260619 08:59:56.868374 19899 heartbeater.cc:464] Registering TS with master...
I20260619 08:59:56.868538 19899 heartbeater.cc:511] Master 127.0.98.62:39751 requested a full tablet report, sending...
I20260619 08:59:56.868630 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 19770
I20260619 08:59:56.868734 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-1/wal/instance
I20260619 08:59:56.868922 19574 ts_manager.cc:194] Registered new tserver with Master: ef22e9e54f654292bfdbc6dd0b6dc09d (127.0.98.2:46633)
I20260619 08:59:56.868947 392 external_mini_cluster.cc:1428] Setting key f923281185451624d2a3c078cc30042b
I20260619 08:59:56.869365 19574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33313
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 08:59:56.892431 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39751
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:59:56.969883 19904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:59:56.970088 19904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:59:56.970108 19904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:59:56.971189 19904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:59:56.971221 19904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:59:56.971244 19904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 08:59:56.972661 19904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:59:56.972834 19904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:59:56.973019 19904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:59:56.977787 19909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:56.977880 19910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:56.977787 19912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:56.978000 19904 server_base.cc:1034] running on GCE node
I20260619 08:59:56.978188 19904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:59:56.978376 19904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:59:56.979518 19904 hybrid_clock.cc:648] HybridClock initialized: now 1781859596979508 us; error 25 us; skew 500 ppm
Jun 19 08:59:56 dist-test-slave-h4g5 krb5kdc[19529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:59:56.982028 19904 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 08:59:56.982437 19904 webserver.cc:460] Webserver started at http://127.0.98.3:36691/ using document root <none> and password file <none>
I20260619 08:59:56.982628 19904 fs_manager.cc:359] Metadata directory not provided
I20260619 08:59:56.982674 19904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:59:56.982798 19904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:59:56.983814 19904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-2/data/instance:
uuid: "d38cca4b21e04e0d9622e8d72994b903"
format_stamp: "Formatted at 2026-06-19 08:59:56 on dist-test-slave-h4g5"
server_key: "4c530bf0619aa4e606b8ce1dcd5cd5f4"
server_key_iv: "3227bfa6a104338e38e5e5430b75bf5b"
server_key_version: "encryptionkey@0"
I20260619 08:59:56.984202 19904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d38cca4b21e04e0d9622e8d72994b903"
format_stamp: "Formatted at 2026-06-19 08:59:56 on dist-test-slave-h4g5"
server_key: "4c530bf0619aa4e606b8ce1dcd5cd5f4"
server_key_iv: "3227bfa6a104338e38e5e5430b75bf5b"
server_key_version: "encryptionkey@0"
I20260619 08:59:56.985460 19904 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:59:56.986150 19919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:56.986294 19904 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:59:56.986377 19904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-2/wal
uuid: "d38cca4b21e04e0d9622e8d72994b903"
format_stamp: "Formatted at 2026-06-19 08:59:56 on dist-test-slave-h4g5"
server_key: "4c530bf0619aa4e606b8ce1dcd5cd5f4"
server_key_iv: "3227bfa6a104338e38e5e5430b75bf5b"
server_key_version: "encryptionkey@0"
I20260619 08:59:56.986430 19904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:59:57.005709 19904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:59:57.007592 19904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:59:57.007701 19904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:59:57.007890 19904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:59:57.008198 19904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:59:57.008234 19904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:57.008255 19904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:59:57.008277 19904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:57.013475 19904 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35721
I20260619 08:59:57.013547 20032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35721 every 8 connection(s)
I20260619 08:59:57.013847 19904 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
I20260619 08:59:57.016881 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 19904
I20260619 08:59:57.016981 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-2/wal/instance
I20260619 08:59:57.017151 392 external_mini_cluster.cc:1428] Setting key 667921da4bb08ecc2c92e437e776ffde
Jun 19 08:59:57 dist-test-slave-h4g5 krb5kdc[19529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 08:59:57.022688 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:57.014538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53615 (local address 127.0.98.62:39751)
0619 08:59:57.014598 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:57.014600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:57.014780 (+ 180us) server_negotiation.cc:407] Connection header received
0619 08:59:57.015675 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:57.015675 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:57.015686 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:57.015736 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:57.016387 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.016989 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.017999 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.018194 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.018250 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:57.020260 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:57.020261 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:57.020262 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:59:57.020269 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:57.021454 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:57.021829 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:57.021831 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:57.021831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:57.021886 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:57.022097 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:57.022098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:57.022098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:57.022208 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:57.022261 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:57.022517 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 08:59:57.022595 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 08:59:57.022887 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:57.014522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39751 (local address 127.0.98.3:53615)
0619 08:59:57.014664 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:57.014700 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:57.015587 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:57.015793 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:57.015797 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:57.016054 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:59:57.016304 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:57.016309 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.017078 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.017078 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:57.017913 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:57.017915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.018263 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.018264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:57.018389 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:57.018798 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 08:59:57.018805 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:59:57.020177 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:59:57.021542 (+ 1365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:59:57.021549 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:59:57.021552 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:59:57.021762 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:59:57.021939 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 08:59:57.021940 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 08:59:57.021940 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 08:59:57.022043 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 08:59:57.022262 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:59:57.022268 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:59:57.022402 (+ 134us) client_negotiation.cc:769] Sending connection context
0619 08:59:57.022488 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 08:59:57.022646 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":83,"threads_started":1}
I20260619 08:59:57.023304 20033 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39751
I20260619 08:59:57.023398 20033 heartbeater.cc:464] Registering TS with master...
I20260619 08:59:57.023579 20033 heartbeater.cc:511] Master 127.0.98.62:39751 requested a full tablet report, sending...
I20260619 08:59:57.023908 19574 ts_manager.cc:194] Registered new tserver with Master: d38cca4b21e04e0d9622e8d72994b903 (127.0.98.3:35721)
I20260619 08:59:57.024299 19574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53615
I20260619 08:59:57.028841 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:59:57.035503 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:57.030173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55694 (local address 127.0.98.62:39751)
0619 08:59:57.030217 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:57.030219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:57.030348 (+ 129us) server_negotiation.cc:407] Connection header received
0619 08:59:57.030381 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:57.030382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:57.030390 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:57.030433 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:57.030941 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.031448 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.032167 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.032358 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.032408 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:57.033467 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:57.033468 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:57.033468 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:59:57.033473 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:57.034591 (+ 1118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:57.034895 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:57.034896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:57.034897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:57.034957 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:57.035099 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:57.035100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:57.035100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:57.035235 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:57.035305 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:57.035381 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 08:59:57.035439 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 08:59:57.037079 19574 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55694:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:59:57.038654 19574 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:59:57.044950 20044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:57.040936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53258 (local address 127.0.98.2:46633)
0619 08:59:57.041099 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:57.041104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:57.041301 (+ 197us) server_negotiation.cc:407] Connection header received
0619 08:59:57.041371 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:57.041373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:57.041388 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:57.041463 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:59:57.041691 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.042538 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.043981 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.044538 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.044573 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:57.044611 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:57.044815 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 08:59:57.044854 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":88,"threads_started":1}
I20260619 08:59:57.044991 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:57.040749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35721 (local address 127.0.0.1:48338)
0619 08:59:57.041122 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:57.041162 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:57.041235 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:57.041481 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:57.041484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:57.041486 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:59:57.041560 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:57.041563 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.042979 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.042980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:57.043884 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:57.043887 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.044768 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.044769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:57.044821 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:57.044823 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:59:57.044870 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 08:59:57.044927 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":4,"thread_start_us":58,"threads_started":1}
I20260619 08:59:57.045210 20042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:57.040924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42901 (local address 127.0.0.1:57394)
0619 08:59:57.041331 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:57.041339 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:57.041386 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:57.041634 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:57.041635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:57.041636 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:59:57.041671 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:57.041672 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.042930 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.042931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:57.044106 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:57.044108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.044986 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.044987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:57.045045 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:57.045050 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 08:59:57.045108 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 08:59:57.045146 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":4,"thread_start_us":64,"threads_started":1}
I20260619 08:59:57.045260 20045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:57.041071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57394 (local address 127.0.98.1:42901)
0619 08:59:57.041230 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:57.041234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:57.041384 (+ 150us) server_negotiation.cc:407] Connection header received
0619 08:59:57.041459 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:57.041462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:57.041478 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:57.041549 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:59:57.041742 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.042826 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.044164 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.044847 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.044904 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:57.044953 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:57.045137 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 08:59:57.045184 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":73,"threads_started":1}
I20260619 08:59:57.045307 20043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:57.040750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46633 (local address 127.0.0.1:53258)
0619 08:59:57.041231 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:57.041241 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:57.041301 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:57.041555 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:57.041556 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:57.041558 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:59:57.041607 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:57.041608 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.042629 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.042630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:57.043889 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:57.043891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.044656 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.044657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:57.044709 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:57.044718 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 08:59:57.045168 (+ 450us) client_negotiation.cc:240] Negotiation successful
0619 08:59:57.045207 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":3,"thread_start_us":63,"threads_started":1}
I20260619 08:59:57.045655 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:57.041060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48338 (local address 127.0.98.3:35721)
0619 08:59:57.041313 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:57.041317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:57.041326 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:59:57.041365 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:57.041368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:57.041403 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:57.041478 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:59:57.041639 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.042822 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.043977 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.044682 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.044760 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:57.044818 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:57.044882 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 08:59:57.044920 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":86,"threads_started":1}
I20260619 08:59:57.045943 19833 tablet_service.cc:1467] Processing CreateTablet for tablet 578022b82b2c4fc0a41ac34902ea997c (DEFAULT_TABLE table=test-table [id=41533bf103e84677a026e08666320503]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:59:57.045944 19699 tablet_service.cc:1467] Processing CreateTablet for tablet 578022b82b2c4fc0a41ac34902ea997c (DEFAULT_TABLE table=test-table [id=41533bf103e84677a026e08666320503]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:59:57.046290 19699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 578022b82b2c4fc0a41ac34902ea997c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:59:57.046290 19833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 578022b82b2c4fc0a41ac34902ea997c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:59:57.046270 19967 tablet_service.cc:1467] Processing CreateTablet for tablet 578022b82b2c4fc0a41ac34902ea997c (DEFAULT_TABLE table=test-table [id=41533bf103e84677a026e08666320503]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:59:57.046527 19967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 578022b82b2c4fc0a41ac34902ea997c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:59:57.049093 20047 tablet_bootstrap.cc:492] T 578022b82b2c4fc0a41ac34902ea997c P d38cca4b21e04e0d9622e8d72994b903: Bootstrap starting.
I20260619 08:59:57.049402 20048 tablet_bootstrap.cc:492] T 578022b82b2c4fc0a41ac34902ea997c P 4835edbbf16f44b1b76dd3cfeb7d1b54: Bootstrap starting.
I20260619 08:59:57.049947 20047 tablet_bootstrap.cc:654] T 578022b82b2c4fc0a41ac34902ea997c P d38cca4b21e04e0d9622e8d72994b903: Neither blocks nor log segments found. Creating new log.
I20260619 08:59:57.049988 20048 tablet_bootstrap.cc:654] T 578022b82b2c4fc0a41ac34902ea997c P 4835edbbf16f44b1b76dd3cfeb7d1b54: Neither blocks nor log segments found. Creating new log.
I20260619 08:59:57.050537 20048 log.cc:824] T 578022b82b2c4fc0a41ac34902ea997c P 4835edbbf16f44b1b76dd3cfeb7d1b54: Log is configured to *not* fsync() on all Append() calls
I20260619 08:59:57.050537 20047 log.cc:824] T 578022b82b2c4fc0a41ac34902ea997c P d38cca4b21e04e0d9622e8d72994b903: Log is configured to *not* fsync() on all Append() calls
I20260619 08:59:57.050856 20049 tablet_bootstrap.cc:492] T 578022b82b2c4fc0a41ac34902ea997c P ef22e9e54f654292bfdbc6dd0b6dc09d: Bootstrap starting.
I20260619 08:59:57.051285 20048 tablet_bootstrap.cc:492] T 578022b82b2c4fc0a41ac34902ea997c P 4835edbbf16f44b1b76dd3cfeb7d1b54: No bootstrap required, opened a new log
I20260619 08:59:57.051362 20048 ts_tablet_manager.cc:1397] T 578022b82b2c4fc0a41ac34902ea997c P 4835edbbf16f44b1b76dd3cfeb7d1b54: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:59:57.051375 20047 tablet_bootstrap.cc:492] T 578022b82b2c4fc0a41ac34902ea997c P d38cca4b21e04e0d9622e8d72994b903: No bootstrap required, opened a new log
I20260619 08:59:57.051421 20049 tablet_bootstrap.cc:654] T 578022b82b2c4fc0a41ac34902ea997c P ef22e9e54f654292bfdbc6dd0b6dc09d: Neither blocks nor log segments found. Creating new log.
I20260619 08:59:57.051445 20047 ts_tablet_manager.cc:1397] T 578022b82b2c4fc0a41ac34902ea997c P d38cca4b21e04e0d9622e8d72994b903: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 08:59:57.051923 20049 log.cc:824] T 578022b82b2c4fc0a41ac34902ea997c P ef22e9e54f654292bfdbc6dd0b6dc09d: Log is configured to *not* fsync() on all Append() calls
I20260619 08:59:57.052810 20049 tablet_bootstrap.cc:492] T 578022b82b2c4fc0a41ac34902ea997c P ef22e9e54f654292bfdbc6dd0b6dc09d: No bootstrap required, opened a new log
I20260619 08:59:57.052897 20049 ts_tablet_manager.cc:1397] T 578022b82b2c4fc0a41ac34902ea997c P ef22e9e54f654292bfdbc6dd0b6dc09d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:59:57.053123 20047 raft_consensus.cc:348] T 578022b82b2c4fc0a41ac34902ea997c P d38cca4b21e04e0d9622e8d72994b903 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38cca4b21e04e0d9622e8d72994b903" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35721 } } peers { permanent_uuid: "ef22e9e54f654292bfdbc6dd0b6dc09d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46633 } } peers { permanent_uuid: "4835edbbf16f44b1b76dd3cfeb7d1b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42901 } }
I20260619 08:59:57.053167 20048 raft_consensus.cc:348] T 578022b82b2c4fc0a41ac34902ea997c P 4835edbbf16f44b1b76dd3cfeb7d1b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38cca4b21e04e0d9622e8d72994b903" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35721 } } peers { permanent_uuid: "ef22e9e54f654292bfdbc6dd0b6dc09d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46633 } } peers { permanent_uuid: "4835edbbf16f44b1b76dd3cfeb7d1b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42901 } }
I20260619 08:59:57.053293 20048 raft_consensus.cc:374] T 578022b82b2c4fc0a41ac34902ea997c P 4835edbbf16f44b1b76dd3cfeb7d1b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:59:57.053293 20047 raft_consensus.cc:374] T 578022b82b2c4fc0a41ac34902ea997c P d38cca4b21e04e0d9622e8d72994b903 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:59:57.053323 20048 raft_consensus.cc:729] T 578022b82b2c4fc0a41ac34902ea997c P 4835edbbf16f44b1b76dd3cfeb7d1b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4835edbbf16f44b1b76dd3cfeb7d1b54, State: Initialized, Role: FOLLOWER
I20260619 08:59:57.053328 20047 raft_consensus.cc:729] T 578022b82b2c4fc0a41ac34902ea997c P d38cca4b21e04e0d9622e8d72994b903 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d38cca4b21e04e0d9622e8d72994b903, State: Initialized, Role: FOLLOWER
I20260619 08:59:57.053442 20047 consensus_queue.cc:260] T 578022b82b2c4fc0a41ac34902ea997c P d38cca4b21e04e0d9622e8d72994b903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38cca4b21e04e0d9622e8d72994b903" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35721 } } peers { permanent_uuid: "ef22e9e54f654292bfdbc6dd0b6dc09d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46633 } } peers { permanent_uuid: "4835edbbf16f44b1b76dd3cfeb7d1b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42901 } }
I20260619 08:59:57.053460 20048 consensus_queue.cc:260] T 578022b82b2c4fc0a41ac34902ea997c P 4835edbbf16f44b1b76dd3cfeb7d1b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38cca4b21e04e0d9622e8d72994b903" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35721 } } peers { permanent_uuid: "ef22e9e54f654292bfdbc6dd0b6dc09d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46633 } } peers { permanent_uuid: "4835edbbf16f44b1b76dd3cfeb7d1b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42901 } }
I20260619 08:59:57.053714 20047 ts_tablet_manager.cc:1428] T 578022b82b2c4fc0a41ac34902ea997c P d38cca4b21e04e0d9622e8d72994b903: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 08:59:57.053740 20048 ts_tablet_manager.cc:1428] T 578022b82b2c4fc0a41ac34902ea997c P 4835edbbf16f44b1b76dd3cfeb7d1b54: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 08:59:57.053751 20033 heartbeater.cc:503] Master 127.0.98.62:39751 was elected leader, sending a full tablet report...
I20260619 08:59:57.053851 19765 heartbeater.cc:503] Master 127.0.98.62:39751 was elected leader, sending a full tablet report...
I20260619 08:59:57.054628 20049 raft_consensus.cc:348] T 578022b82b2c4fc0a41ac34902ea997c P ef22e9e54f654292bfdbc6dd0b6dc09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38cca4b21e04e0d9622e8d72994b903" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35721 } } peers { permanent_uuid: "ef22e9e54f654292bfdbc6dd0b6dc09d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46633 } } peers { permanent_uuid: "4835edbbf16f44b1b76dd3cfeb7d1b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42901 } }
I20260619 08:59:57.054786 20049 raft_consensus.cc:374] T 578022b82b2c4fc0a41ac34902ea997c P ef22e9e54f654292bfdbc6dd0b6dc09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:59:57.054831 20049 raft_consensus.cc:729] T 578022b82b2c4fc0a41ac34902ea997c P ef22e9e54f654292bfdbc6dd0b6dc09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef22e9e54f654292bfdbc6dd0b6dc09d, State: Initialized, Role: FOLLOWER
I20260619 08:59:57.054953 20049 consensus_queue.cc:260] T 578022b82b2c4fc0a41ac34902ea997c P ef22e9e54f654292bfdbc6dd0b6dc09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38cca4b21e04e0d9622e8d72994b903" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35721 } } peers { permanent_uuid: "ef22e9e54f654292bfdbc6dd0b6dc09d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46633 } } peers { permanent_uuid: "4835edbbf16f44b1b76dd3cfeb7d1b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42901 } }
I20260619 08:59:57.055197 20049 ts_tablet_manager.cc:1428] T 578022b82b2c4fc0a41ac34902ea997c P ef22e9e54f654292bfdbc6dd0b6dc09d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 08:59:57.055213 19899 heartbeater.cc:503] Master 127.0.98.62:39751 was elected leader, sending a full tablet report...
W20260619 08:59:57.109858 19900 tablet.cc:2369] T 578022b82b2c4fc0a41ac34902ea997c P ef22e9e54f654292bfdbc6dd0b6dc09d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 08:59:57.211585 19766 tablet.cc:2369] T 578022b82b2c4fc0a41ac34902ea997c P 4835edbbf16f44b1b76dd3cfeb7d1b54: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:59:57.235571 20055 raft_consensus.cc:482] T 578022b82b2c4fc0a41ac34902ea997c P ef22e9e54f654292bfdbc6dd0b6dc09d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:59:57.235697 20055 raft_consensus.cc:504] T 578022b82b2c4fc0a41ac34902ea997c P ef22e9e54f654292bfdbc6dd0b6dc09d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38cca4b21e04e0d9622e8d72994b903" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35721 } } peers { permanent_uuid: "ef22e9e54f654292bfdbc6dd0b6dc09d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46633 } } peers { permanent_uuid: "4835edbbf16f44b1b76dd3cfeb7d1b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42901 } }
I20260619 08:59:57.236043 20055 leader_election.cc:288] T 578022b82b2c4fc0a41ac34902ea997c P ef22e9e54f654292bfdbc6dd0b6dc09d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d38cca4b21e04e0d9622e8d72994b903 (127.0.98.3:35721), 4835edbbf16f44b1b76dd3cfeb7d1b54 (127.0.98.1:42901)
I20260619 08:59:57.239830 20045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:57.236323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46059 (local address 127.0.98.1:42901)
0619 08:59:57.236408 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:57.236410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:57.236421 (+ 11us) server_negotiation.cc:407] Connection header received
0619 08:59:57.236479 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:57.236480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:57.236492 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:57.236539 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:59:57.236694 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.237448 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.238905 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.239466 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.239539 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:57.239550 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:57.239724 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 08:59:57.239758 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 08:59:57.239843 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:57.236344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52589 (local address 127.0.98.3:35721)
0619 08:59:57.236387 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:57.236389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:57.236550 (+ 161us) server_negotiation.cc:407] Connection header received
0619 08:59:57.236583 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:57.236584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:57.236592 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:57.236633 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:59:57.236754 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.237445 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.238905 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.239469 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.239539 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:57.239549 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:57.239724 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 08:59:57.239760 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 08:59:57.239850 19901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:57.236274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42901 (local address 127.0.98.2:46059)
0619 08:59:57.236332 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:57.236363 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:57.236410 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:57.236561 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:57.236562 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:57.236564 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:59:57.236633 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:57.236634 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.237533 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.237534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:57.238810 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:57.238814 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.239554 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.239554 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:57.239652 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:57.239658 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 08:59:57.239715 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 08:59:57.239758 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":6}
I20260619 08:59:57.240057 20056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:57.236274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35721 (local address 127.0.98.2:52589)
0619 08:59:57.236489 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:57.236496 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:57.236545 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:57.236647 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:57.236647 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:57.236648 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:59:57.236700 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:57.236702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.237533 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.237534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:57.238810 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:57.238814 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.239554 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.239554 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:57.239652 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:57.239658 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 08:59:57.239715 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 08:59:57.239757 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":137,"thread_start_us":104,"threads_started":1}
I20260619 08:59:57.240203 19719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "578022b82b2c4fc0a41ac34902ea997c" candidate_uuid: "ef22e9e54f654292bfdbc6dd0b6dc09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4835edbbf16f44b1b76dd3cfeb7d1b54" is_pre_election: true
I20260619 08:59:57.240363 19719 raft_consensus.cc:2454] T 578022b82b2c4fc0a41ac34902ea997c P 4835edbbf16f44b1b76dd3cfeb7d1b54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef22e9e54f654292bfdbc6dd0b6dc09d in term 0.
I20260619 08:59:57.240360 19987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "578022b82b2c4fc0a41ac34902ea997c" candidate_uuid: "ef22e9e54f654292bfdbc6dd0b6dc09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d38cca4b21e04e0d9622e8d72994b903" is_pre_election: true
I20260619 08:59:57.240533 19987 raft_consensus.cc:2454] T 578022b82b2c4fc0a41ac34902ea997c P d38cca4b21e04e0d9622e8d72994b903 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef22e9e54f654292bfdbc6dd0b6dc09d in term 0.
I20260619 08:59:57.240562 19786 leader_election.cc:302] T 578022b82b2c4fc0a41ac34902ea997c P ef22e9e54f654292bfdbc6dd0b6dc09d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4835edbbf16f44b1b76dd3cfeb7d1b54, ef22e9e54f654292bfdbc6dd0b6dc09d; no voters:
I20260619 08:59:57.240679 20055 raft_consensus.cc:2781] T 578022b82b2c4fc0a41ac34902ea997c P ef22e9e54f654292bfdbc6dd0b6dc09d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:59:57.240747 20055 raft_consensus.cc:482] T 578022b82b2c4fc0a41ac34902ea997c P ef22e9e54f654292bfdbc6dd0b6dc09d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:59:57.240763 20055 raft_consensus.cc:3037] T 578022b82b2c4fc0a41ac34902ea997c P ef22e9e54f654292bfdbc6dd0b6dc09d [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:59:57.241384 20055 raft_consensus.cc:504] T 578022b82b2c4fc0a41ac34902ea997c P ef22e9e54f654292bfdbc6dd0b6dc09d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38cca4b21e04e0d9622e8d72994b903" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35721 } } peers { permanent_uuid: "ef22e9e54f654292bfdbc6dd0b6dc09d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46633 } } peers { permanent_uuid: "4835edbbf16f44b1b76dd3cfeb7d1b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42901 } }
I20260619 08:59:57.241513 20055 leader_election.cc:288] T 578022b82b2c4fc0a41ac34902ea997c P ef22e9e54f654292bfdbc6dd0b6dc09d [CANDIDATE]: Term 1 election: Requested vote from peers d38cca4b21e04e0d9622e8d72994b903 (127.0.98.3:35721), 4835edbbf16f44b1b76dd3cfeb7d1b54 (127.0.98.1:42901)
I20260619 08:59:57.241636 19987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "578022b82b2c4fc0a41ac34902ea997c" candidate_uuid: "ef22e9e54f654292bfdbc6dd0b6dc09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d38cca4b21e04e0d9622e8d72994b903"
I20260619 08:59:57.241657 19719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "578022b82b2c4fc0a41ac34902ea997c" candidate_uuid: "ef22e9e54f654292bfdbc6dd0b6dc09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4835edbbf16f44b1b76dd3cfeb7d1b54"
I20260619 08:59:57.241703 19987 raft_consensus.cc:3037] T 578022b82b2c4fc0a41ac34902ea997c P d38cca4b21e04e0d9622e8d72994b903 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:59:57.241720 19719 raft_consensus.cc:3037] T 578022b82b2c4fc0a41ac34902ea997c P 4835edbbf16f44b1b76dd3cfeb7d1b54 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:59:57.242355 19987 raft_consensus.cc:2454] T 578022b82b2c4fc0a41ac34902ea997c P d38cca4b21e04e0d9622e8d72994b903 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef22e9e54f654292bfdbc6dd0b6dc09d in term 1.
I20260619 08:59:57.242370 19719 raft_consensus.cc:2454] T 578022b82b2c4fc0a41ac34902ea997c P 4835edbbf16f44b1b76dd3cfeb7d1b54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef22e9e54f654292bfdbc6dd0b6dc09d in term 1.
I20260619 08:59:57.242512 19787 leader_election.cc:302] T 578022b82b2c4fc0a41ac34902ea997c P ef22e9e54f654292bfdbc6dd0b6dc09d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d38cca4b21e04e0d9622e8d72994b903, ef22e9e54f654292bfdbc6dd0b6dc09d; no voters:
I20260619 08:59:57.242592 20055 raft_consensus.cc:2781] T 578022b82b2c4fc0a41ac34902ea997c P ef22e9e54f654292bfdbc6dd0b6dc09d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:59:57.242743 20055 raft_consensus.cc:686] T 578022b82b2c4fc0a41ac34902ea997c P ef22e9e54f654292bfdbc6dd0b6dc09d [term 1 LEADER]: Becoming Leader. State: Replica: ef22e9e54f654292bfdbc6dd0b6dc09d, State: Running, Role: LEADER
I20260619 08:59:57.242836 20055 consensus_queue.cc:237] T 578022b82b2c4fc0a41ac34902ea997c P ef22e9e54f654292bfdbc6dd0b6dc09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38cca4b21e04e0d9622e8d72994b903" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35721 } } peers { permanent_uuid: "ef22e9e54f654292bfdbc6dd0b6dc09d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46633 } } peers { permanent_uuid: "4835edbbf16f44b1b76dd3cfeb7d1b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42901 } }
I20260619 08:59:57.243634 19574 catalog_manager.cc:5495] T 578022b82b2c4fc0a41ac34902ea997c P ef22e9e54f654292bfdbc6dd0b6dc09d reported cstate change: term changed from 0 to 1, leader changed from <none> to ef22e9e54f654292bfdbc6dd0b6dc09d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ef22e9e54f654292bfdbc6dd0b6dc09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d38cca4b21e04e0d9622e8d72994b903" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef22e9e54f654292bfdbc6dd0b6dc09d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4835edbbf16f44b1b76dd3cfeb7d1b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42901 } health_report { overall_health: UNKNOWN } } }
W20260619 08:59:57.264390 20034 tablet.cc:2369] T 578022b82b2c4fc0a41ac34902ea997c P d38cca4b21e04e0d9622e8d72994b903: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:59:57.270313 20044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:57.268069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53272 (local address 127.0.98.2:46633)
0619 08:59:57.268152 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:57.268154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:57.268164 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:59:57.268202 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:57.268203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:57.268223 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:57.268280 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:59:57.268399 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.268853 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.269586 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.269768 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.269821 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:57.269984 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:59:57.270085 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:59:57.270156 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:57.270192 (+ 36us) server_negotiation.cc:299] Negotiation successful
0619 08:59:57.270212 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 08:59:57.272120 19719 raft_consensus.cc:1261] T 578022b82b2c4fc0a41ac34902ea997c P 4835edbbf16f44b1b76dd3cfeb7d1b54 [term 1 FOLLOWER]: Refusing update from remote peer ef22e9e54f654292bfdbc6dd0b6dc09d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:59:57.272121 19987 raft_consensus.cc:1261] T 578022b82b2c4fc0a41ac34902ea997c P d38cca4b21e04e0d9622e8d72994b903 [term 1 FOLLOWER]: Refusing update from remote peer ef22e9e54f654292bfdbc6dd0b6dc09d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:59:57.272454 20057 consensus_queue.cc:1035] T 578022b82b2c4fc0a41ac34902ea997c P ef22e9e54f654292bfdbc6dd0b6dc09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d38cca4b21e04e0d9622e8d72994b903" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:59:57.272586 20055 consensus_queue.cc:1035] T 578022b82b2c4fc0a41ac34902ea997c P ef22e9e54f654292bfdbc6dd0b6dc09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4835edbbf16f44b1b76dd3cfeb7d1b54" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:59:57.274827 20060 mvcc.cc:205] Tried to move back new op lower bound from 7298496910423662592 to 7298496910306947072. Current Snapshot: MvccSnapshot[applied={T|T < 7298496910423662592}]
I20260619 08:59:57.274972 20061 mvcc.cc:205] Tried to move back new op lower bound from 7298496910423662592 to 7298496910306947072. Current Snapshot: MvccSnapshot[applied={T|T < 7298496910423662592}]
I20260619 08:59:57.275684 20062 mvcc.cc:205] Tried to move back new op lower bound from 7298496910423662592 to 7298496910306947072. Current Snapshot: MvccSnapshot[applied={T|T < 7298496910423662592 or (T in {7298496910423662592})}]
I20260619 08:59:57.277200 19573 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:59:57.277298 19573 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:59:57.277990 19573 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4fb1772c6ba4c26925d8204a64a926a: Sending DeleteTablet for 3 replicas of tablet 578022b82b2c4fc0a41ac34902ea997c
I20260619 08:59:57.278264 19833 tablet_service.cc:1514] Processing DeleteTablet for tablet 578022b82b2c4fc0a41ac34902ea997c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:59:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53258
I20260619 08:59:57.278291 19967 tablet_service.cc:1514] Processing DeleteTablet for tablet 578022b82b2c4fc0a41ac34902ea997c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:59:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48338
I20260619 08:59:57.278321 19699 tablet_service.cc:1514] Processing DeleteTablet for tablet 578022b82b2c4fc0a41ac34902ea997c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:59:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57394
I20260619 08:59:57.278529 20072 tablet_replica.cc:330] T 578022b82b2c4fc0a41ac34902ea997c P 4835edbbf16f44b1b76dd3cfeb7d1b54: stopping tablet replica
I20260619 08:59:57.278628 20072 raft_consensus.cc:2229] T 578022b82b2c4fc0a41ac34902ea997c P 4835edbbf16f44b1b76dd3cfeb7d1b54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:59:57.278735 20072 raft_consensus.cc:2258] T 578022b82b2c4fc0a41ac34902ea997c P 4835edbbf16f44b1b76dd3cfeb7d1b54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:59:57.278826 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 19636
W20260619 08:59:57.283329 19558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42901 (error 108)
I20260619 08:59:57.283864 20071 tablet_replica.cc:330] T 578022b82b2c4fc0a41ac34902ea997c P d38cca4b21e04e0d9622e8d72994b903: stopping tablet replica
I20260619 08:59:57.283859 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 19770
I20260619 08:59:57.283913 20043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:57.283790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42901 (local address 127.0.0.1:57398)
0619 08:59:57.283841 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:57.283879 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 08:59:57.284010 20071 raft_consensus.cc:2229] T 578022b82b2c4fc0a41ac34902ea997c P d38cca4b21e04e0d9622e8d72994b903 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260619 08:59:57.284034 19558 catalog_manager.cc:4584] TS 4835edbbf16f44b1b76dd3cfeb7d1b54 (127.0.98.1:42901): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 578022b82b2c4fc0a41ac34902ea997c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42901: connect: Connection refused (error 111)
I20260619 08:59:57.284135 20071 raft_consensus.cc:2258] T 578022b82b2c4fc0a41ac34902ea997c P d38cca4b21e04e0d9622e8d72994b903 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:59:57.284678 20071 ts_tablet_manager.cc:1905] T 578022b82b2c4fc0a41ac34902ea997c P d38cca4b21e04e0d9622e8d72994b903: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:59:57.286036 20071 ts_tablet_manager.cc:1918] T 578022b82b2c4fc0a41ac34902ea997c P d38cca4b21e04e0d9622e8d72994b903: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:59:57.286094 20071 log.cc:1196] T 578022b82b2c4fc0a41ac34902ea997c P d38cca4b21e04e0d9622e8d72994b903: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859361540516-392-0/minicluster-data/ts-2/wal/wals/578022b82b2c4fc0a41ac34902ea997c
I20260619 08:59:57.286296 20071 ts_tablet_manager.cc:1939] T 578022b82b2c4fc0a41ac34902ea997c P d38cca4b21e04e0d9622e8d72994b903: Deleting consensus metadata
I20260619 08:59:57.286686 19559 catalog_manager.cc:4841] TS d38cca4b21e04e0d9622e8d72994b903 (127.0.98.3:35721): tablet 578022b82b2c4fc0a41ac34902ea997c (table test-table [id=41533bf103e84677a026e08666320503]) successfully deleted
I20260619 08:59:57.289000 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 19904
I20260619 08:59:57.289496 20043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:57.288899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46633 (local address 127.0.0.1:53284)
0619 08:59:57.289433 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:57.289466 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511}
I20260619 08:59:57.292881 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 19542
2026-06-19T08:59:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1968 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T08:59:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:59:57Z Disabled control of system clock
I20260619 08:59:57.321370 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43881
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 08:59:57.401381 20077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:59:57.401610 20077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:59:57.401633 20077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:59:57.402832 20077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:59:57.402880 20077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:59:57.402894 20077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:59:57.402910 20077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:59:57.402938 20077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:59:57.404338 20077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43881
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:59:57.404587 20077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:59:57.404896 20077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:59:57.409642 20085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:57.409708 20083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:57.409642 20082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:57.409669 20077 server_base.cc:1034] running on GCE node
I20260619 08:59:57.410178 20077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:59:57.410491 20077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:59:57.411639 20077 hybrid_clock.cc:648] HybridClock initialized: now 1781859597411621 us; error 33 us; skew 500 ppm
I20260619 08:59:57.412997 20077 webserver.cc:460] Webserver started at http://127.0.98.62:37127/ using document root <none> and password file <none>
I20260619 08:59:57.413234 20077 fs_manager.cc:359] Metadata directory not provided
I20260619 08:59:57.413281 20077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:59:57.413411 20077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:59:57.414469 20077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "4b11b5539d2a4bf0bfbeff2e1f00a972"
format_stamp: "Formatted at 2026-06-19 08:59:57 on dist-test-slave-h4g5"
server_key: "79f1c1c8829c6c8d0af0567d6f557e3a"
server_key_iv: "029c04f7c503d40e728acf9641a75b8b"
server_key_version: "encryptionkey@0"
I20260619 08:59:57.414837 20077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "4b11b5539d2a4bf0bfbeff2e1f00a972"
format_stamp: "Formatted at 2026-06-19 08:59:57 on dist-test-slave-h4g5"
server_key: "79f1c1c8829c6c8d0af0567d6f557e3a"
server_key_iv: "029c04f7c503d40e728acf9641a75b8b"
server_key_version: "encryptionkey@0"
I20260619 08:59:57.416049 20077 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:59:57.416731 20091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:57.416944 20077 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:59:57.417024 20077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "4b11b5539d2a4bf0bfbeff2e1f00a972"
format_stamp: "Formatted at 2026-06-19 08:59:57 on dist-test-slave-h4g5"
server_key: "79f1c1c8829c6c8d0af0567d6f557e3a"
server_key_iv: "029c04f7c503d40e728acf9641a75b8b"
server_key_version: "encryptionkey@0"
I20260619 08:59:57.417093 20077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:59:57.431989 20077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:59:57.432281 20077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:59:57.432412 20077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:59:57.435963 20077 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43881
I20260619 08:59:57.436012 20143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43881 every 8 connection(s)
I20260619 08:59:57.436371 20077 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:59:57.437312 20144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:59:57.439589 20144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b11b5539d2a4bf0bfbeff2e1f00a972: Bootstrap starting.
I20260619 08:59:57.440232 20144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b11b5539d2a4bf0bfbeff2e1f00a972: Neither blocks nor log segments found. Creating new log.
I20260619 08:59:57.440517 20144 log.cc:824] T 00000000000000000000000000000000 P 4b11b5539d2a4bf0bfbeff2e1f00a972: Log is configured to *not* fsync() on all Append() calls
I20260619 08:59:57.441157 20144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b11b5539d2a4bf0bfbeff2e1f00a972: No bootstrap required, opened a new log
I20260619 08:59:57.442695 20144 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b11b5539d2a4bf0bfbeff2e1f00a972 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b11b5539d2a4bf0bfbeff2e1f00a972" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43881 } }
I20260619 08:59:57.442830 20144 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b11b5539d2a4bf0bfbeff2e1f00a972 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:59:57.442863 20144 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b11b5539d2a4bf0bfbeff2e1f00a972 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b11b5539d2a4bf0bfbeff2e1f00a972, State: Initialized, Role: FOLLOWER
I20260619 08:59:57.442996 20144 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b11b5539d2a4bf0bfbeff2e1f00a972 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b11b5539d2a4bf0bfbeff2e1f00a972" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43881 } }
I20260619 08:59:57.443073 20144 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b11b5539d2a4bf0bfbeff2e1f00a972 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:59:57.443102 20144 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b11b5539d2a4bf0bfbeff2e1f00a972 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:59:57.443148 20144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b11b5539d2a4bf0bfbeff2e1f00a972 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:59:57.443643 20144 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b11b5539d2a4bf0bfbeff2e1f00a972 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b11b5539d2a4bf0bfbeff2e1f00a972" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43881 } }
I20260619 08:59:57.443733 20144 leader_election.cc:302] T 00000000000000000000000000000000 P 4b11b5539d2a4bf0bfbeff2e1f00a972 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b11b5539d2a4bf0bfbeff2e1f00a972; no voters:
I20260619 08:59:57.443886 20144 leader_election.cc:288] T 00000000000000000000000000000000 P 4b11b5539d2a4bf0bfbeff2e1f00a972 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:59:57.443928 20147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b11b5539d2a4bf0bfbeff2e1f00a972 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:59:57.444108 20147 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b11b5539d2a4bf0bfbeff2e1f00a972 [term 1 LEADER]: Becoming Leader. State: Replica: 4b11b5539d2a4bf0bfbeff2e1f00a972, State: Running, Role: LEADER
I20260619 08:59:57.444154 20144 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b11b5539d2a4bf0bfbeff2e1f00a972 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:59:57.444208 20147 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b11b5539d2a4bf0bfbeff2e1f00a972 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b11b5539d2a4bf0bfbeff2e1f00a972" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43881 } }
I20260619 08:59:57.444600 20148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b11b5539d2a4bf0bfbeff2e1f00a972 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b11b5539d2a4bf0bfbeff2e1f00a972" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b11b5539d2a4bf0bfbeff2e1f00a972" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43881 } } }
I20260619 08:59:57.444679 20149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b11b5539d2a4bf0bfbeff2e1f00a972 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b11b5539d2a4bf0bfbeff2e1f00a972. Latest consensus state: current_term: 1 leader_uuid: "4b11b5539d2a4bf0bfbeff2e1f00a972" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b11b5539d2a4bf0bfbeff2e1f00a972" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43881 } } }
I20260619 08:59:57.444762 20148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b11b5539d2a4bf0bfbeff2e1f00a972 [sys.catalog]: This master's current role is: LEADER
I20260619 08:59:57.444769 20149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b11b5539d2a4bf0bfbeff2e1f00a972 [sys.catalog]: This master's current role is: LEADER
I20260619 08:59:57.445147 20156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:59:57.445297 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 20077
I20260619 08:59:57.445407 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 08:59:57.445582 392 external_mini_cluster.cc:1428] Setting key 53dbebe2a8b646a720da7c57457f5410
I20260619 08:59:57.445672 20156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:59:57.448423 20156 catalog_manager.cc:1347] Generated new cluster ID: de4f275a4ab54ae58d95f26be4d951cd
I20260619 08:59:57.448479 20156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:59:57.452250 20166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:57.446192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55512 (local address 127.0.98.62:43881)
0619 08:59:57.446773 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:57.446782 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:57.446810 (+ 28us) server_negotiation.cc:407] Connection header received
0619 08:59:57.446929 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:57.446936 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:57.447003 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:57.447141 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:57.447340 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.448736 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.449889 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.450498 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.450966 (+ 468us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:57.451331 (+ 365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:57.451335 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:57.451335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:59:57.451550 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:57.451590 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:59:57.451604 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:57.451897 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:57.451941 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 08:59:57.452020 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":67,"threads_started":1}
I20260619 08:59:57.457650 20156 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:59:57.458125 20156 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:59:57.462766 20156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b11b5539d2a4bf0bfbeff2e1f00a972: Generated new TSK 0
I20260619 08:59:57.462918 20156 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:59:57.470953 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43881
--builtin_ntp_servers=127.0.98.20:45437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 08:59:57.549945 20168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:59:57.550151 20168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:59:57.550179 20168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:59:57.551311 20168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:59:57.551352 20168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:59:57.551384 20168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 08:59:57.552755 20168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:59:57.552979 20168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:59:57.553249 20168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:59:57.558094 20176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:57.558132 20173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:57.558094 20174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:57.558312 20168 server_base.cc:1034] running on GCE node
I20260619 08:59:57.558516 20168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:59:57.558777 20168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:59:57.559948 20168 hybrid_clock.cc:648] HybridClock initialized: now 1781859597559916 us; error 48 us; skew 500 ppm
I20260619 08:59:57.561352 20168 webserver.cc:460] Webserver started at http://127.0.98.1:41357/ using document root <none> and password file <none>
I20260619 08:59:57.561596 20168 fs_manager.cc:359] Metadata directory not provided
I20260619 08:59:57.561650 20168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:59:57.561769 20168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:59:57.563107 20168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-0/data/instance:
uuid: "3bbfeeaeaf6943dc8a7380aaec616d08"
format_stamp: "Formatted at 2026-06-19 08:59:57 on dist-test-slave-h4g5"
server_key: "3b6f94b9fdc0ea61092e9248fb6d23e8"
server_key_iv: "678cbf8d5fcbf548bf5207b86116f7de"
server_key_version: "encryptionkey@0"
I20260619 08:59:57.563462 20168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3bbfeeaeaf6943dc8a7380aaec616d08"
format_stamp: "Formatted at 2026-06-19 08:59:57 on dist-test-slave-h4g5"
server_key: "3b6f94b9fdc0ea61092e9248fb6d23e8"
server_key_iv: "678cbf8d5fcbf548bf5207b86116f7de"
server_key_version: "encryptionkey@0"
I20260619 08:59:57.564982 20168 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:59:57.565837 20182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:57.566062 20168 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:59:57.566160 20168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-0/wal
uuid: "3bbfeeaeaf6943dc8a7380aaec616d08"
format_stamp: "Formatted at 2026-06-19 08:59:57 on dist-test-slave-h4g5"
server_key: "3b6f94b9fdc0ea61092e9248fb6d23e8"
server_key_iv: "678cbf8d5fcbf548bf5207b86116f7de"
server_key_version: "encryptionkey@0"
I20260619 08:59:57.566237 20168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:59:57.582005 20168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:59:57.582286 20168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:59:57.582388 20168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:59:57.582584 20168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:59:57.582842 20168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:59:57.582871 20168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:57.582896 20168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:59:57.582911 20168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:57.588371 20168 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38395
I20260619 08:59:57.588428 20295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38395 every 8 connection(s)
I20260619 08:59:57.588726 20168 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-0/data/info.pb
I20260619 08:59:57.593477 20166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:57.589407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48239 (local address 127.0.98.62:43881)
0619 08:59:57.589456 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:57.589458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:57.589699 (+ 241us) server_negotiation.cc:407] Connection header received
0619 08:59:57.590184 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:57.590185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:57.590196 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:57.590259 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:57.590955 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.591439 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.592396 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.592572 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.592618 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:57.593281 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:57.593282 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:57.593282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:59:57.593288 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:57.593301 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:59:57.593305 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:57.593343 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:57.593393 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 08:59:57.593414 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 08:59:57.593691 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:57.589394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43881 (local address 127.0.98.1:48239)
0619 08:59:57.589599 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:57.589633 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:57.590112 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:57.590352 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:57.590357 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:57.590367 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:59:57.590856 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:57.590861 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.591511 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.591512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:57.592340 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:57.592342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.592625 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.592626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:57.592725 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:57.593166 (+ 441us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:59:57.593166 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:59:57.593194 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:59:57.593215 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:59:57.593342 (+ 127us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:59:57.593345 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:59:57.593347 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:59:57.593390 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 08:59:57.593475 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":94,"threads_started":1}
I20260619 08:59:57.594125 20296 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43881
I20260619 08:59:57.594203 20296 heartbeater.cc:464] Registering TS with master...
I20260619 08:59:57.594364 20296 heartbeater.cc:511] Master 127.0.98.62:43881 requested a full tablet report, sending...
I20260619 08:59:57.594812 20108 ts_manager.cc:194] Registered new tserver with Master: 3bbfeeaeaf6943dc8a7380aaec616d08 (127.0.98.1:38395)
I20260619 08:59:57.595366 20108 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:48239
I20260619 08:59:57.595911 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 20168
I20260619 08:59:57.596012 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-0/wal/instance
I20260619 08:59:57.596230 392 external_mini_cluster.cc:1428] Setting key 1145be93d7eac04b2304b862d14709c2
I20260619 08:59:57.597023 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43881
--builtin_ntp_servers=127.0.98.20:45437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 08:59:57.675292 20299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:59:57.675493 20299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:59:57.675514 20299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:59:57.676669 20299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:59:57.676707 20299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:59:57.676731 20299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 08:59:57.678011 20299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:59:57.678191 20299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:59:57.678435 20299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:59:57.683207 20305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:57.683203 20304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:57.683497 20299 server_base.cc:1034] running on GCE node
W20260619 08:59:57.683225 20307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:57.683738 20299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:59:57.683969 20299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:59:57.685127 20299 hybrid_clock.cc:648] HybridClock initialized: now 1781859597685113 us; error 29 us; skew 500 ppm
I20260619 08:59:57.686268 20299 webserver.cc:460] Webserver started at http://127.0.98.2:45987/ using document root <none> and password file <none>
I20260619 08:59:57.686494 20299 fs_manager.cc:359] Metadata directory not provided
I20260619 08:59:57.686544 20299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:59:57.686671 20299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:59:57.687709 20299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-1/data/instance:
uuid: "312645e759b34be4ab40c3db24069aa7"
format_stamp: "Formatted at 2026-06-19 08:59:57 on dist-test-slave-h4g5"
server_key: "b98c7438fc8469706d742adaf40fb51c"
server_key_iv: "e27f69f8a6b692f6fec3ab8cf2ff2f85"
server_key_version: "encryptionkey@0"
I20260619 08:59:57.688046 20299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-1/wal/instance:
uuid: "312645e759b34be4ab40c3db24069aa7"
format_stamp: "Formatted at 2026-06-19 08:59:57 on dist-test-slave-h4g5"
server_key: "b98c7438fc8469706d742adaf40fb51c"
server_key_iv: "e27f69f8a6b692f6fec3ab8cf2ff2f85"
server_key_version: "encryptionkey@0"
I20260619 08:59:57.689252 20299 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:59:57.689955 20313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:57.690126 20299 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 08:59:57.690205 20299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-1/wal
uuid: "312645e759b34be4ab40c3db24069aa7"
format_stamp: "Formatted at 2026-06-19 08:59:57 on dist-test-slave-h4g5"
server_key: "b98c7438fc8469706d742adaf40fb51c"
server_key_iv: "e27f69f8a6b692f6fec3ab8cf2ff2f85"
server_key_version: "encryptionkey@0"
I20260619 08:59:57.690272 20299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:59:57.712126 20299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:59:57.712461 20299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:59:57.712612 20299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:59:57.712838 20299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:59:57.713168 20299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:59:57.713203 20299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:57.713231 20299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:59:57.713248 20299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:57.718590 20299 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40199
I20260619 08:59:57.718631 20426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40199 every 8 connection(s)
I20260619 08:59:57.718935 20299 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-1/data/info.pb
I20260619 08:59:57.721276 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 20299
I20260619 08:59:57.721367 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-1/wal/instance
I20260619 08:59:57.721517 392 external_mini_cluster.cc:1428] Setting key 93a65e12d6ae435a475e00f0de259f36
I20260619 08:59:57.722414 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43881
--builtin_ntp_servers=127.0.98.20:45437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 08:59:57.724287 20166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:57.719626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57091 (local address 127.0.98.62:43881)
0619 08:59:57.719678 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:57.719681 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:57.719916 (+ 235us) server_negotiation.cc:407] Connection header received
0619 08:59:57.720410 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:57.720411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:57.720420 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:57.720476 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:57.721099 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.721688 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.722856 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.723142 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.723208 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:57.723977 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:57.723979 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:57.723979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:59:57.723983 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:57.723998 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:59:57.724001 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:57.724119 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:57.724198 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 08:59:57.724227 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 08:59:57.724511 20429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:57.719595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43881 (local address 127.0.98.2:57091)
0619 08:59:57.719796 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:57.719837 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:57.720336 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:57.720535 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:57.720545 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:57.720553 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:59:57.721031 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:57.721036 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.721775 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.721776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:57.722780 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:57.722782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.723217 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.723218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:57.723318 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:57.723828 (+ 510us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:59:57.723829 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:59:57.723868 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:59:57.723896 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:59:57.724119 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:59:57.724124 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:59:57.724126 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 08:59:57.724193 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 08:59:57.724299 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":70,"threads_started":1}
I20260619 08:59:57.724933 20427 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43881
I20260619 08:59:57.725031 20427 heartbeater.cc:464] Registering TS with master...
I20260619 08:59:57.725211 20427 heartbeater.cc:511] Master 127.0.98.62:43881 requested a full tablet report, sending...
I20260619 08:59:57.725530 20108 ts_manager.cc:194] Registered new tserver with Master: 312645e759b34be4ab40c3db24069aa7 (127.0.98.2:40199)
I20260619 08:59:57.725878 20108 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:57091
W20260619 08:59:57.801467 20430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:59:57.801676 20430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:59:57.801695 20430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:59:57.802843 20430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:59:57.802886 20430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:59:57.802911 20430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 08:59:57.804265 20430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:59:57.804481 20430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:59:57.804749 20430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:59:57.809574 20438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:57.809609 20435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:57.809646 20430 server_base.cc:1034] running on GCE node
W20260619 08:59:57.809561 20436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:57.809908 20430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:59:57.810119 20430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:59:57.811257 20430 hybrid_clock.cc:648] HybridClock initialized: now 1781859597811247 us; error 30 us; skew 500 ppm
I20260619 08:59:57.812453 20430 webserver.cc:460] Webserver started at http://127.0.98.3:40303/ using document root <none> and password file <none>
I20260619 08:59:57.812670 20430 fs_manager.cc:359] Metadata directory not provided
I20260619 08:59:57.812718 20430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:59:57.812840 20430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:59:57.813866 20430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-2/data/instance:
uuid: "f868bc4a526d491a9e7d3582b9e3b810"
format_stamp: "Formatted at 2026-06-19 08:59:57 on dist-test-slave-h4g5"
server_key: "9f776ba2b4855d39cef3d139ef9385b0"
server_key_iv: "0b4e2ed8b42a3e6fd71cc61d7f165696"
server_key_version: "encryptionkey@0"
I20260619 08:59:57.814198 20430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f868bc4a526d491a9e7d3582b9e3b810"
format_stamp: "Formatted at 2026-06-19 08:59:57 on dist-test-slave-h4g5"
server_key: "9f776ba2b4855d39cef3d139ef9385b0"
server_key_iv: "0b4e2ed8b42a3e6fd71cc61d7f165696"
server_key_version: "encryptionkey@0"
I20260619 08:59:57.815368 20430 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:59:57.816102 20444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:57.816254 20430 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:59:57.816335 20430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-2/wal
uuid: "f868bc4a526d491a9e7d3582b9e3b810"
format_stamp: "Formatted at 2026-06-19 08:59:57 on dist-test-slave-h4g5"
server_key: "9f776ba2b4855d39cef3d139ef9385b0"
server_key_iv: "0b4e2ed8b42a3e6fd71cc61d7f165696"
server_key_version: "encryptionkey@0"
I20260619 08:59:57.816401 20430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:59:57.833479 20430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:59:57.833801 20430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:59:57.833933 20430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:59:57.834141 20430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:59:57.834458 20430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:59:57.834491 20430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:57.834520 20430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:59:57.834554 20430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:57.839989 20430 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38543
I20260619 08:59:57.840049 20557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38543 every 8 connection(s)
I20260619 08:59:57.840358 20430 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-2/data/info.pb
I20260619 08:59:57.845261 20166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:57.840986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43163 (local address 127.0.98.62:43881)
0619 08:59:57.841032 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:57.841033 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:57.841275 (+ 242us) server_negotiation.cc:407] Connection header received
0619 08:59:57.841773 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:57.841774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:57.841787 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:57.841837 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:57.842515 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.843005 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.843984 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.844232 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.844283 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:57.844996 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:57.844998 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:57.844998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:59:57.845003 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:57.845018 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:59:57.845021 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:57.845067 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:57.845156 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 08:59:57.845184 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 08:59:57.845438 20560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:57.840987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43881 (local address 127.0.98.3:43163)
0619 08:59:57.841154 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:57.841189 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:57.841675 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:57.841909 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:57.841912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:57.841923 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0619 08:59:57.842422 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:57.842428 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.843085 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.843086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:57.843898 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:57.843901 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.844308 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.844309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:57.844430 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:57.844872 (+ 442us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 08:59:57.844872 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 08:59:57.844899 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 08:59:57.844921 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 08:59:57.845088 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 08:59:57.845093 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 08:59:57.845096 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 08:59:57.845153 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 08:59:57.845251 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":73,"threads_started":1}
I20260619 08:59:57.845850 20558 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43881
I20260619 08:59:57.845932 20558 heartbeater.cc:464] Registering TS with master...
I20260619 08:59:57.846098 20558 heartbeater.cc:511] Master 127.0.98.62:43881 requested a full tablet report, sending...
I20260619 08:59:57.846431 20108 ts_manager.cc:194] Registered new tserver with Master: f868bc4a526d491a9e7d3582b9e3b810 (127.0.98.3:38543)
I20260619 08:59:57.846776 20108 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:43163
I20260619 08:59:57.847434 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 20430
I20260619 08:59:57.847527 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-2/wal/instance
I20260619 08:59:57.847700 392 external_mini_cluster.cc:1428] Setting key b55d41889eaf7713e4d9fb13c5b9af9a
I20260619 08:59:57.848428 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:59:57.852419 20166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:57.849719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55526 (local address 127.0.98.62:43881)
0619 08:59:57.849793 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:57.849794 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:57.849869 (+ 75us) server_negotiation.cc:407] Connection header received
0619 08:59:57.849933 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:57.849934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:57.849945 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:57.849998 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:57.850189 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.850643 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.851374 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.851556 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.851607 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:57.852199 (+ 592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:57.852200 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:57.852201 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 08:59:57.852205 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:57.852219 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 08:59:57.852223 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:57.852263 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:57.852319 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 08:59:57.852341 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 08:59:57.853729 20108 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55526:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 08:59:57.855198 20108 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:59:57.861058 20566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:57.857097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40199 (local address 127.0.0.1:60406)
0619 08:59:57.857216 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:57.857254 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:57.857313 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:57.857920 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:57.857921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:57.857924 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:59:57.857967 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:57.857969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.859226 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.859228 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:57.860056 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:57.860058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.860775 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.860776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:57.860855 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:57.860869 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 08:59:57.860932 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 08:59:57.860982 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":52,"threads_started":1}
I20260619 08:59:57.861079 20570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:57.857506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60406 (local address 127.0.98.2:40199)
0619 08:59:57.857759 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:57.857763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:57.857772 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:59:57.857807 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:57.857809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:57.857822 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:57.858078 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:59:57.858137 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.859162 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.860165 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.860690 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.860756 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:57.860848 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:57.860956 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 08:59:57.861002 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":75,"threads_started":1}
I20260619 08:59:57.861220 20568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:57.857288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38543 (local address 127.0.0.1:52144)
0619 08:59:57.857534 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:57.857543 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:57.857589 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:57.857906 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:57.857907 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:57.857909 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:59:57.857944 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:57.857946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.858797 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.858798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:57.859974 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:57.859977 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.861002 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.861003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:57.861065 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:57.861070 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 08:59:57.861101 (+ 31us) client_negotiation.cc:240] Negotiation successful
0619 08:59:57.861139 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":74,"threads_started":1}
I20260619 08:59:57.861418 20567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:57.857099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38395 (local address 127.0.0.1:34946)
0619 08:59:57.857219 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:57.857259 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:57.857313 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:57.857682 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:57.857686 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:57.857688 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:59:57.857750 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:57.857752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.859103 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.859104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:57.860153 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:57.860155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.861220 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.861221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:57.861277 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:57.861281 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:59:57.861328 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 08:59:57.861363 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
I20260619 08:59:57.861644 20571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:57.857569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52144 (local address 127.0.98.3:38543)
0619 08:59:57.857725 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:57.857731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:57.857739 (+ 8us) server_negotiation.cc:407] Connection header received
0619 08:59:57.857797 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:57.857799 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:57.857815 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:57.858011 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:59:57.858064 (+ 53us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.858723 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.860052 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.860892 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.861387 (+ 495us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:57.861504 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:57.861530 (+ 26us) server_negotiation.cc:299] Negotiation successful
0619 08:59:57.861570 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
I20260619 08:59:57.861662 20569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:57.857327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34946 (local address 127.0.98.1:38395)
0619 08:59:57.857522 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:57.857527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:57.857537 (+ 10us) server_negotiation.cc:407] Connection header received
0619 08:59:57.857576 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:57.857579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:57.857593 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:57.857673 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:59:57.857869 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.859019 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.860210 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.861045 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.861105 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:57.861187 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:57.861552 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 08:59:57.861591 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":95,"threads_started":1}
I20260619 08:59:57.862073 20361 tablet_service.cc:1467] Processing CreateTablet for tablet 01633b103a214f7fa89f2449f2b5313d (DEFAULT_TABLE table=test-table [id=360088ea320449a1a61593c8d34765d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:59:57.862272 20230 tablet_service.cc:1467] Processing CreateTablet for tablet 01633b103a214f7fa89f2449f2b5313d (DEFAULT_TABLE table=test-table [id=360088ea320449a1a61593c8d34765d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:59:57.862272 20492 tablet_service.cc:1467] Processing CreateTablet for tablet 01633b103a214f7fa89f2449f2b5313d (DEFAULT_TABLE table=test-table [id=360088ea320449a1a61593c8d34765d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:59:57.862411 20361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01633b103a214f7fa89f2449f2b5313d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:59:57.862592 20230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01633b103a214f7fa89f2449f2b5313d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:59:57.862592 20492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01633b103a214f7fa89f2449f2b5313d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:59:57.864904 20572 tablet_bootstrap.cc:492] T 01633b103a214f7fa89f2449f2b5313d P 312645e759b34be4ab40c3db24069aa7: Bootstrap starting.
I20260619 08:59:57.865497 20572 tablet_bootstrap.cc:654] T 01633b103a214f7fa89f2449f2b5313d P 312645e759b34be4ab40c3db24069aa7: Neither blocks nor log segments found. Creating new log.
I20260619 08:59:57.865792 20574 tablet_bootstrap.cc:492] T 01633b103a214f7fa89f2449f2b5313d P 3bbfeeaeaf6943dc8a7380aaec616d08: Bootstrap starting.
I20260619 08:59:57.865945 20572 log.cc:824] T 01633b103a214f7fa89f2449f2b5313d P 312645e759b34be4ab40c3db24069aa7: Log is configured to *not* fsync() on all Append() calls
I20260619 08:59:57.866370 20574 tablet_bootstrap.cc:654] T 01633b103a214f7fa89f2449f2b5313d P 3bbfeeaeaf6943dc8a7380aaec616d08: Neither blocks nor log segments found. Creating new log.
I20260619 08:59:57.866672 20572 tablet_bootstrap.cc:492] T 01633b103a214f7fa89f2449f2b5313d P 312645e759b34be4ab40c3db24069aa7: No bootstrap required, opened a new log
I20260619 08:59:57.866667 20574 log.cc:824] T 01633b103a214f7fa89f2449f2b5313d P 3bbfeeaeaf6943dc8a7380aaec616d08: Log is configured to *not* fsync() on all Append() calls
I20260619 08:59:57.866739 20572 ts_tablet_manager.cc:1397] T 01633b103a214f7fa89f2449f2b5313d P 312645e759b34be4ab40c3db24069aa7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:59:57.866967 20573 tablet_bootstrap.cc:492] T 01633b103a214f7fa89f2449f2b5313d P f868bc4a526d491a9e7d3582b9e3b810: Bootstrap starting.
I20260619 08:59:57.867300 20574 tablet_bootstrap.cc:492] T 01633b103a214f7fa89f2449f2b5313d P 3bbfeeaeaf6943dc8a7380aaec616d08: No bootstrap required, opened a new log
I20260619 08:59:57.867355 20574 ts_tablet_manager.cc:1397] T 01633b103a214f7fa89f2449f2b5313d P 3bbfeeaeaf6943dc8a7380aaec616d08: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:59:57.867565 20573 tablet_bootstrap.cc:654] T 01633b103a214f7fa89f2449f2b5313d P f868bc4a526d491a9e7d3582b9e3b810: Neither blocks nor log segments found. Creating new log.
I20260619 08:59:57.867856 20573 log.cc:824] T 01633b103a214f7fa89f2449f2b5313d P f868bc4a526d491a9e7d3582b9e3b810: Log is configured to *not* fsync() on all Append() calls
I20260619 08:59:57.868577 20573 tablet_bootstrap.cc:492] T 01633b103a214f7fa89f2449f2b5313d P f868bc4a526d491a9e7d3582b9e3b810: No bootstrap required, opened a new log
I20260619 08:59:57.868637 20573 ts_tablet_manager.cc:1397] T 01633b103a214f7fa89f2449f2b5313d P f868bc4a526d491a9e7d3582b9e3b810: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:59:57.869166 20572 raft_consensus.cc:348] T 01633b103a214f7fa89f2449f2b5313d P 312645e759b34be4ab40c3db24069aa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312645e759b34be4ab40c3db24069aa7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40199 } } peers { permanent_uuid: "f868bc4a526d491a9e7d3582b9e3b810" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38543 } } peers { permanent_uuid: "3bbfeeaeaf6943dc8a7380aaec616d08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38395 } }
I20260619 08:59:57.869343 20572 raft_consensus.cc:374] T 01633b103a214f7fa89f2449f2b5313d P 312645e759b34be4ab40c3db24069aa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:59:57.869374 20572 raft_consensus.cc:729] T 01633b103a214f7fa89f2449f2b5313d P 312645e759b34be4ab40c3db24069aa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 312645e759b34be4ab40c3db24069aa7, State: Initialized, Role: FOLLOWER
I20260619 08:59:57.869406 20574 raft_consensus.cc:348] T 01633b103a214f7fa89f2449f2b5313d P 3bbfeeaeaf6943dc8a7380aaec616d08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312645e759b34be4ab40c3db24069aa7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40199 } } peers { permanent_uuid: "f868bc4a526d491a9e7d3582b9e3b810" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38543 } } peers { permanent_uuid: "3bbfeeaeaf6943dc8a7380aaec616d08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38395 } }
I20260619 08:59:57.869505 20572 consensus_queue.cc:260] T 01633b103a214f7fa89f2449f2b5313d P 312645e759b34be4ab40c3db24069aa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312645e759b34be4ab40c3db24069aa7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40199 } } peers { permanent_uuid: "f868bc4a526d491a9e7d3582b9e3b810" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38543 } } peers { permanent_uuid: "3bbfeeaeaf6943dc8a7380aaec616d08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38395 } }
I20260619 08:59:57.869557 20574 raft_consensus.cc:374] T 01633b103a214f7fa89f2449f2b5313d P 3bbfeeaeaf6943dc8a7380aaec616d08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:59:57.869598 20574 raft_consensus.cc:729] T 01633b103a214f7fa89f2449f2b5313d P 3bbfeeaeaf6943dc8a7380aaec616d08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bbfeeaeaf6943dc8a7380aaec616d08, State: Initialized, Role: FOLLOWER
I20260619 08:59:57.869699 20574 consensus_queue.cc:260] T 01633b103a214f7fa89f2449f2b5313d P 3bbfeeaeaf6943dc8a7380aaec616d08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312645e759b34be4ab40c3db24069aa7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40199 } } peers { permanent_uuid: "f868bc4a526d491a9e7d3582b9e3b810" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38543 } } peers { permanent_uuid: "3bbfeeaeaf6943dc8a7380aaec616d08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38395 } }
I20260619 08:59:57.869776 20572 ts_tablet_manager.cc:1428] T 01633b103a214f7fa89f2449f2b5313d P 312645e759b34be4ab40c3db24069aa7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:59:57.869902 20427 heartbeater.cc:503] Master 127.0.98.62:43881 was elected leader, sending a full tablet report...
I20260619 08:59:57.869925 20574 ts_tablet_manager.cc:1428] T 01633b103a214f7fa89f2449f2b5313d P 3bbfeeaeaf6943dc8a7380aaec616d08: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:59:57.870011 20296 heartbeater.cc:503] Master 127.0.98.62:43881 was elected leader, sending a full tablet report...
I20260619 08:59:57.870965 20573 raft_consensus.cc:348] T 01633b103a214f7fa89f2449f2b5313d P f868bc4a526d491a9e7d3582b9e3b810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312645e759b34be4ab40c3db24069aa7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40199 } } peers { permanent_uuid: "f868bc4a526d491a9e7d3582b9e3b810" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38543 } } peers { permanent_uuid: "3bbfeeaeaf6943dc8a7380aaec616d08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38395 } }
I20260619 08:59:57.871120 20573 raft_consensus.cc:374] T 01633b103a214f7fa89f2449f2b5313d P f868bc4a526d491a9e7d3582b9e3b810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:59:57.871160 20573 raft_consensus.cc:729] T 01633b103a214f7fa89f2449f2b5313d P f868bc4a526d491a9e7d3582b9e3b810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f868bc4a526d491a9e7d3582b9e3b810, State: Initialized, Role: FOLLOWER
I20260619 08:59:57.871250 20573 consensus_queue.cc:260] T 01633b103a214f7fa89f2449f2b5313d P f868bc4a526d491a9e7d3582b9e3b810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312645e759b34be4ab40c3db24069aa7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40199 } } peers { permanent_uuid: "f868bc4a526d491a9e7d3582b9e3b810" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38543 } } peers { permanent_uuid: "3bbfeeaeaf6943dc8a7380aaec616d08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38395 } }
I20260619 08:59:57.871464 20573 ts_tablet_manager.cc:1428] T 01633b103a214f7fa89f2449f2b5313d P f868bc4a526d491a9e7d3582b9e3b810: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:59:57.871542 20558 heartbeater.cc:503] Master 127.0.98.62:43881 was elected leader, sending a full tablet report...
W20260619 08:59:57.969447 20428 tablet.cc:2369] T 01633b103a214f7fa89f2449f2b5313d P 312645e759b34be4ab40c3db24069aa7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:59:57.977823 20580 raft_consensus.cc:482] T 01633b103a214f7fa89f2449f2b5313d P f868bc4a526d491a9e7d3582b9e3b810 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:59:57.977938 20580 raft_consensus.cc:504] T 01633b103a214f7fa89f2449f2b5313d P f868bc4a526d491a9e7d3582b9e3b810 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312645e759b34be4ab40c3db24069aa7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40199 } } peers { permanent_uuid: "f868bc4a526d491a9e7d3582b9e3b810" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38543 } } peers { permanent_uuid: "3bbfeeaeaf6943dc8a7380aaec616d08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38395 } }
I20260619 08:59:57.978278 20580 leader_election.cc:288] T 01633b103a214f7fa89f2449f2b5313d P f868bc4a526d491a9e7d3582b9e3b810 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 312645e759b34be4ab40c3db24069aa7 (127.0.98.2:40199), 3bbfeeaeaf6943dc8a7380aaec616d08 (127.0.98.1:38395)
I20260619 08:59:57.981235 20569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:57.978539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58333 (local address 127.0.98.1:38395)
0619 08:59:57.978583 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:57.978585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:57.978597 (+ 12us) server_negotiation.cc:407] Connection header received
0619 08:59:57.978683 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:57.978683 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:57.978693 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:57.978746 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:59:57.978943 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.979516 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.980555 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.980907 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.980959 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:57.980968 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:57.981126 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 08:59:57.981167 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 08:59:57.981247 20560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:57.978481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38395 (local address 127.0.98.3:58333)
0619 08:59:57.978538 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:57.978550 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:57.978611 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:57.978774 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:57.978775 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:57.978777 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:59:57.978862 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:57.978864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.979595 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.979596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:57.980474 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:57.980477 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.980982 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.980983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:57.981049 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:57.981053 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:59:57.981104 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 08:59:57.981141 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 08:59:57.981278 20570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:57.978562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36351 (local address 127.0.98.2:40199)
0619 08:59:57.978633 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:57.978635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:57.978919 (+ 284us) server_negotiation.cc:407] Connection header received
0619 08:59:57.978954 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:57.978955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:57.978966 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:57.979013 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 08:59:57.979149 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.979617 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.980590 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.980957 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.981036 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:57.981045 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:57.981183 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 08:59:57.981220 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 08:59:57.981477 20581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:57.978491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40199 (local address 127.0.98.3:36351)
0619 08:59:57.978848 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:57.978856 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 08:59:57.978904 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 08:59:57.979022 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 08:59:57.979023 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 08:59:57.979024 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 08:59:57.979088 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:57.979090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.979701 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.979701 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:57.980501 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 08:59:57.980503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 08:59:57.981044 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 08:59:57.981044 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 08:59:57.981110 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:57.981114 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 08:59:57.981165 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 08:59:57.981196 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":194,"thread_start_us":96,"threads_started":1}
I20260619 08:59:57.981678 20250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01633b103a214f7fa89f2449f2b5313d" candidate_uuid: "f868bc4a526d491a9e7d3582b9e3b810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bbfeeaeaf6943dc8a7380aaec616d08" is_pre_election: true
I20260619 08:59:57.981827 20250 raft_consensus.cc:2454] T 01633b103a214f7fa89f2449f2b5313d P 3bbfeeaeaf6943dc8a7380aaec616d08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f868bc4a526d491a9e7d3582b9e3b810 in term 0.
I20260619 08:59:57.981949 20381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01633b103a214f7fa89f2449f2b5313d" candidate_uuid: "f868bc4a526d491a9e7d3582b9e3b810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "312645e759b34be4ab40c3db24069aa7" is_pre_election: true
I20260619 08:59:57.982107 20381 raft_consensus.cc:2454] T 01633b103a214f7fa89f2449f2b5313d P 312645e759b34be4ab40c3db24069aa7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f868bc4a526d491a9e7d3582b9e3b810 in term 0.
I20260619 08:59:57.982091 20448 leader_election.cc:302] T 01633b103a214f7fa89f2449f2b5313d P f868bc4a526d491a9e7d3582b9e3b810 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bbfeeaeaf6943dc8a7380aaec616d08, f868bc4a526d491a9e7d3582b9e3b810; no voters:
I20260619 08:59:57.982280 20580 raft_consensus.cc:2781] T 01633b103a214f7fa89f2449f2b5313d P f868bc4a526d491a9e7d3582b9e3b810 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:59:57.982352 20580 raft_consensus.cc:482] T 01633b103a214f7fa89f2449f2b5313d P f868bc4a526d491a9e7d3582b9e3b810 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:59:57.982375 20580 raft_consensus.cc:3037] T 01633b103a214f7fa89f2449f2b5313d P f868bc4a526d491a9e7d3582b9e3b810 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:59:57.983055 20580 raft_consensus.cc:504] T 01633b103a214f7fa89f2449f2b5313d P f868bc4a526d491a9e7d3582b9e3b810 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312645e759b34be4ab40c3db24069aa7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40199 } } peers { permanent_uuid: "f868bc4a526d491a9e7d3582b9e3b810" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38543 } } peers { permanent_uuid: "3bbfeeaeaf6943dc8a7380aaec616d08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38395 } }
I20260619 08:59:57.983232 20580 leader_election.cc:288] T 01633b103a214f7fa89f2449f2b5313d P f868bc4a526d491a9e7d3582b9e3b810 [CANDIDATE]: Term 1 election: Requested vote from peers 312645e759b34be4ab40c3db24069aa7 (127.0.98.2:40199), 3bbfeeaeaf6943dc8a7380aaec616d08 (127.0.98.1:38395)
I20260619 08:59:57.983312 20381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01633b103a214f7fa89f2449f2b5313d" candidate_uuid: "f868bc4a526d491a9e7d3582b9e3b810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "312645e759b34be4ab40c3db24069aa7"
I20260619 08:59:57.983390 20381 raft_consensus.cc:3037] T 01633b103a214f7fa89f2449f2b5313d P 312645e759b34be4ab40c3db24069aa7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:59:57.984069 20381 raft_consensus.cc:2454] T 01633b103a214f7fa89f2449f2b5313d P 312645e759b34be4ab40c3db24069aa7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f868bc4a526d491a9e7d3582b9e3b810 in term 1.
I20260619 08:59:57.984239 20250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01633b103a214f7fa89f2449f2b5313d" candidate_uuid: "f868bc4a526d491a9e7d3582b9e3b810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bbfeeaeaf6943dc8a7380aaec616d08"
I20260619 08:59:57.984277 20447 leader_election.cc:302] T 01633b103a214f7fa89f2449f2b5313d P f868bc4a526d491a9e7d3582b9e3b810 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 312645e759b34be4ab40c3db24069aa7, f868bc4a526d491a9e7d3582b9e3b810; no voters:
I20260619 08:59:57.984333 20250 raft_consensus.cc:3037] T 01633b103a214f7fa89f2449f2b5313d P 3bbfeeaeaf6943dc8a7380aaec616d08 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:59:57.984393 20580 raft_consensus.cc:2781] T 01633b103a214f7fa89f2449f2b5313d P f868bc4a526d491a9e7d3582b9e3b810 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:59:57.984557 20580 raft_consensus.cc:686] T 01633b103a214f7fa89f2449f2b5313d P f868bc4a526d491a9e7d3582b9e3b810 [term 1 LEADER]: Becoming Leader. State: Replica: f868bc4a526d491a9e7d3582b9e3b810, State: Running, Role: LEADER
I20260619 08:59:57.984652 20580 consensus_queue.cc:237] T 01633b103a214f7fa89f2449f2b5313d P f868bc4a526d491a9e7d3582b9e3b810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312645e759b34be4ab40c3db24069aa7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40199 } } peers { permanent_uuid: "f868bc4a526d491a9e7d3582b9e3b810" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38543 } } peers { permanent_uuid: "3bbfeeaeaf6943dc8a7380aaec616d08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38395 } }
I20260619 08:59:57.985275 20250 raft_consensus.cc:2454] T 01633b103a214f7fa89f2449f2b5313d P 3bbfeeaeaf6943dc8a7380aaec616d08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f868bc4a526d491a9e7d3582b9e3b810 in term 1.
I20260619 08:59:57.985476 20107 catalog_manager.cc:5495] T 01633b103a214f7fa89f2449f2b5313d P f868bc4a526d491a9e7d3582b9e3b810 reported cstate change: term changed from 0 to 1, leader changed from <none> to f868bc4a526d491a9e7d3582b9e3b810 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f868bc4a526d491a9e7d3582b9e3b810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "312645e759b34be4ab40c3db24069aa7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f868bc4a526d491a9e7d3582b9e3b810" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bbfeeaeaf6943dc8a7380aaec616d08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38395 } health_report { overall_health: UNKNOWN } } }
I20260619 08:59:58.008569 20571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:58.006294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52156 (local address 127.0.98.3:38543)
0619 08:59:58.006341 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:58.006343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:58.006352 (+ 9us) server_negotiation.cc:407] Connection header received
0619 08:59:58.006411 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:58.006412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:58.006426 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:58.006484 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 08:59:58.006598 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:58.007071 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:58.007855 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:58.008044 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:58.008146 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:58.008267 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 08:59:58.008361 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 08:59:58.008417 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:58.008455 (+ 38us) server_negotiation.cc:299] Negotiation successful
0619 08:59:58.008470 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 08:59:58.010301 20381 raft_consensus.cc:1261] T 01633b103a214f7fa89f2449f2b5313d P 312645e759b34be4ab40c3db24069aa7 [term 1 FOLLOWER]: Refusing update from remote peer f868bc4a526d491a9e7d3582b9e3b810: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:59:58.010300 20250 raft_consensus.cc:1261] T 01633b103a214f7fa89f2449f2b5313d P 3bbfeeaeaf6943dc8a7380aaec616d08 [term 1 FOLLOWER]: Refusing update from remote peer f868bc4a526d491a9e7d3582b9e3b810: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:59:58.010644 20580 consensus_queue.cc:1035] T 01633b103a214f7fa89f2449f2b5313d P f868bc4a526d491a9e7d3582b9e3b810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bbfeeaeaf6943dc8a7380aaec616d08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:59:58.010792 20582 consensus_queue.cc:1035] T 01633b103a214f7fa89f2449f2b5313d P f868bc4a526d491a9e7d3582b9e3b810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "312645e759b34be4ab40c3db24069aa7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:59:58.013217 20587 mvcc.cc:205] Tried to move back new op lower bound from 7298496913447673856 to 7298496913345736704. Current Snapshot: MvccSnapshot[applied={T|T < 7298496913447673856}]
I20260619 08:59:58.015587 20108 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:59:58.015687 20108 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 08:59:58.016371 20108 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b11b5539d2a4bf0bfbeff2e1f00a972: Sending DeleteTablet for 3 replicas of tablet 01633b103a214f7fa89f2449f2b5313d
I20260619 08:59:58.016696 20230 tablet_service.cc:1514] Processing DeleteTablet for tablet 01633b103a214f7fa89f2449f2b5313d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:59:58 UTC) from {username='slave'} at 127.0.0.1:34946
I20260619 08:59:58.016891 20492 tablet_service.cc:1514] Processing DeleteTablet for tablet 01633b103a214f7fa89f2449f2b5313d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:59:58 UTC) from {username='slave'} at 127.0.0.1:52144
I20260619 08:59:58.016942 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 20168
I20260619 08:59:58.017510 20361 tablet_service.cc:1514] Processing DeleteTablet for tablet 01633b103a214f7fa89f2449f2b5313d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 08:59:58 UTC) from {username='slave'} at 127.0.0.1:60406
I20260619 08:59:58.017774 20597 tablet_replica.cc:330] T 01633b103a214f7fa89f2449f2b5313d P 312645e759b34be4ab40c3db24069aa7: stopping tablet replica
I20260619 08:59:58.017860 20597 raft_consensus.cc:2229] T 01633b103a214f7fa89f2449f2b5313d P 312645e759b34be4ab40c3db24069aa7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 08:59:58.018005 20597 raft_consensus.cc:2258] T 01633b103a214f7fa89f2449f2b5313d P 312645e759b34be4ab40c3db24069aa7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:59:58.018448 20597 ts_tablet_manager.cc:1905] T 01633b103a214f7fa89f2449f2b5313d P 312645e759b34be4ab40c3db24069aa7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:59:58.020176 20597 ts_tablet_manager.cc:1918] T 01633b103a214f7fa89f2449f2b5313d P 312645e759b34be4ab40c3db24069aa7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:59:58.020249 20597 log.cc:1196] T 01633b103a214f7fa89f2449f2b5313d P 312645e759b34be4ab40c3db24069aa7: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-1/wal/wals/01633b103a214f7fa89f2449f2b5313d
I20260619 08:59:58.020463 20597 ts_tablet_manager.cc:1939] T 01633b103a214f7fa89f2449f2b5313d P 312645e759b34be4ab40c3db24069aa7: Deleting consensus metadata
I20260619 08:59:58.020879 20094 catalog_manager.cc:4841] TS 312645e759b34be4ab40c3db24069aa7 (127.0.98.2:40199): tablet 01633b103a214f7fa89f2449f2b5313d (table test-table [id=360088ea320449a1a61593c8d34765d2]) successfully deleted
W20260619 08:59:58.022560 20095 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38395 (error 108)
I20260619 08:59:58.023015 20567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:58.022884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38395 (local address 127.0.0.1:34952)
0619 08:59:58.022935 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 08:59:58.022978 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 08:59:58.023033 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 20299
I20260619 08:59:58.023061 20596 tablet_replica.cc:330] T 01633b103a214f7fa89f2449f2b5313d P f868bc4a526d491a9e7d3582b9e3b810: stopping tablet replica
I20260619 08:59:58.023744 20596 raft_consensus.cc:2229] T 01633b103a214f7fa89f2449f2b5313d P f868bc4a526d491a9e7d3582b9e3b810 [term 1 LEADER]: Raft consensus shutting down.
I20260619 08:59:58.023887 20596 raft_consensus.cc:2258] T 01633b103a214f7fa89f2449f2b5313d P f868bc4a526d491a9e7d3582b9e3b810 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 08:59:58.024247 20596 ts_tablet_manager.cc:1905] T 01633b103a214f7fa89f2449f2b5313d P f868bc4a526d491a9e7d3582b9e3b810: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 08:59:58.025513 20596 ts_tablet_manager.cc:1918] T 01633b103a214f7fa89f2449f2b5313d P f868bc4a526d491a9e7d3582b9e3b810: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 08:59:58.025574 20596 log.cc:1196] T 01633b103a214f7fa89f2449f2b5313d P f868bc4a526d491a9e7d3582b9e3b810: Deleting WAL directory at /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859361540516-392-0/minicluster-data/ts-2/wal/wals/01633b103a214f7fa89f2449f2b5313d
I20260619 08:59:58.025782 20596 ts_tablet_manager.cc:1939] T 01633b103a214f7fa89f2449f2b5313d P f868bc4a526d491a9e7d3582b9e3b810: Deleting consensus metadata
I20260619 08:59:58.026124 20094 catalog_manager.cc:4841] TS f868bc4a526d491a9e7d3582b9e3b810 (127.0.98.3:38543): tablet 01633b103a214f7fa89f2449f2b5313d (table test-table [id=360088ea320449a1a61593c8d34765d2]) successfully deleted
I20260619 08:59:58.028561 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 20430
W20260619 08:59:58.028620 20095 catalog_manager.cc:4584] TS 3bbfeeaeaf6943dc8a7380aaec616d08 (127.0.98.1:38395): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 01633b103a214f7fa89f2449f2b5313d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38395: connect: Connection refused (error 111)
I20260619 08:59:58.032953 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 20077
2026-06-19T08:59:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (739 ms)
[----------] 6 tests from EncryptionPolicyTest (7764 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:59:58 dist-test-slave-h4g5 krb5kdc[20601](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:59:58 dist-test-slave-h4g5 krb5kdc[20601](info): set up 2 sockets
Jun 19 08:59:58 dist-test-slave-h4g5 krb5kdc[20601](info): commencing operation
krb5kdc: starting...
W20260619 08:59:58.975791 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 08:59:59 dist-test-slave-h4g5 krb5kdc[20601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T08:59:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:59:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 08:59:59.086706 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34601
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 08:59:59.166435 20614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:59:59.166644 20614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:59:59.166668 20614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:59:59.167814 20614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:59:59.167855 20614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 08:59:59.167868 20614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:59:59.167879 20614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:59:59.167891 20614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:59:59.169361 20614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34601
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:59:59.169579 20614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:59:59.169807 20614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 08:59:59.174400 20614 server_base.cc:1034] running on GCE node
W20260619 08:59:59.174360 20622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:59.174377 20620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:59:59.174369 20619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:59:59.174804 20614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:59:59.175055 20614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:59:59.176194 20614 hybrid_clock.cc:648] HybridClock initialized: now 1781859599176180 us; error 26 us; skew 500 ppm
Jun 19 08:59:59 dist-test-slave-h4g5 krb5kdc[20601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 08:59:59.178558 20614 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 08:59:59.178997 20614 webserver.cc:460] Webserver started at http://127.0.0.1:40847/ using document root <none> and password file <none>
I20260619 08:59:59.179204 20614 fs_manager.cc:359] Metadata directory not provided
I20260619 08:59:59.179250 20614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:59:59.179375 20614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:59:59.180505 20614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "32e1c6d700ed4cd5b38cc78c98920880"
format_stamp: "Formatted at 2026-06-19 08:59:59 on dist-test-slave-h4g5"
server_key: "6ad261c611b5b7802879fff7c06677cd"
server_key_iv: "2117feadc9c129944e2b5d35dadc6e9f"
server_key_version: "encryptionkey@0"
I20260619 08:59:59.180860 20614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "32e1c6d700ed4cd5b38cc78c98920880"
format_stamp: "Formatted at 2026-06-19 08:59:59 on dist-test-slave-h4g5"
server_key: "6ad261c611b5b7802879fff7c06677cd"
server_key_iv: "2117feadc9c129944e2b5d35dadc6e9f"
server_key_version: "encryptionkey@0"
I20260619 08:59:59.182080 20614 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:59:59.182832 20629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:59:59.183009 20614 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:59:59.183090 20614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "32e1c6d700ed4cd5b38cc78c98920880"
format_stamp: "Formatted at 2026-06-19 08:59:59 on dist-test-slave-h4g5"
server_key: "6ad261c611b5b7802879fff7c06677cd"
server_key_iv: "2117feadc9c129944e2b5d35dadc6e9f"
server_key_version: "encryptionkey@0"
I20260619 08:59:59.183153 20614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:59:59.191506 20614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:59:59.193162 20614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:59:59.193285 20614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:59:59.196904 20614 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34601
I20260619 08:59:59.196934 20681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34601 every 8 connection(s)
I20260619 08:59:59.197232 20614 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 08:59:59.197959 20682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:59:59.200052 20682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32e1c6d700ed4cd5b38cc78c98920880: Bootstrap starting.
I20260619 08:59:59.200589 20682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32e1c6d700ed4cd5b38cc78c98920880: Neither blocks nor log segments found. Creating new log.
I20260619 08:59:59.200817 20682 log.cc:824] T 00000000000000000000000000000000 P 32e1c6d700ed4cd5b38cc78c98920880: Log is configured to *not* fsync() on all Append() calls
I20260619 08:59:59.200990 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 20614
I20260619 08:59:59.201099 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 08:59:59.201295 392 external_mini_cluster.cc:1428] Setting key 40f84bec3b9f9daa0253d5ddea4c5de7
I20260619 08:59:59.201542 20682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32e1c6d700ed4cd5b38cc78c98920880: No bootstrap required, opened a new log
I20260619 08:59:59.203711 20682 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32e1c6d700ed4cd5b38cc78c98920880 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e1c6d700ed4cd5b38cc78c98920880" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34601 } }
I20260619 08:59:59.203841 20682 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32e1c6d700ed4cd5b38cc78c98920880 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:59:59.203862 20682 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32e1c6d700ed4cd5b38cc78c98920880 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32e1c6d700ed4cd5b38cc78c98920880, State: Initialized, Role: FOLLOWER
I20260619 08:59:59.203962 20682 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32e1c6d700ed4cd5b38cc78c98920880 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e1c6d700ed4cd5b38cc78c98920880" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34601 } }
I20260619 08:59:59.204030 20682 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32e1c6d700ed4cd5b38cc78c98920880 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:59:59.204058 20682 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32e1c6d700ed4cd5b38cc78c98920880 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:59:59.204123 20682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32e1c6d700ed4cd5b38cc78c98920880 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:59:59.204658 20682 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32e1c6d700ed4cd5b38cc78c98920880 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e1c6d700ed4cd5b38cc78c98920880" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34601 } }
I20260619 08:59:59.204741 20682 leader_election.cc:302] T 00000000000000000000000000000000 P 32e1c6d700ed4cd5b38cc78c98920880 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32e1c6d700ed4cd5b38cc78c98920880; no voters:
I20260619 08:59:59.204880 20682 leader_election.cc:288] T 00000000000000000000000000000000 P 32e1c6d700ed4cd5b38cc78c98920880 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:59:59.204929 20687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32e1c6d700ed4cd5b38cc78c98920880 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:59:59.205116 20682 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32e1c6d700ed4cd5b38cc78c98920880 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:59:59.205121 20687 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32e1c6d700ed4cd5b38cc78c98920880 [term 1 LEADER]: Becoming Leader. State: Replica: 32e1c6d700ed4cd5b38cc78c98920880, State: Running, Role: LEADER
I20260619 08:59:59.205335 20687 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32e1c6d700ed4cd5b38cc78c98920880 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e1c6d700ed4cd5b38cc78c98920880" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34601 } }
I20260619 08:59:59.205694 20688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32e1c6d700ed4cd5b38cc78c98920880 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32e1c6d700ed4cd5b38cc78c98920880" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e1c6d700ed4cd5b38cc78c98920880" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34601 } } }
I20260619 08:59:59.205761 20688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32e1c6d700ed4cd5b38cc78c98920880 [sys.catalog]: This master's current role is: LEADER
I20260619 08:59:59.205952 20689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32e1c6d700ed4cd5b38cc78c98920880 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32e1c6d700ed4cd5b38cc78c98920880. Latest consensus state: current_term: 1 leader_uuid: "32e1c6d700ed4cd5b38cc78c98920880" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e1c6d700ed4cd5b38cc78c98920880" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34601 } } }
I20260619 08:59:59.206239 20689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32e1c6d700ed4cd5b38cc78c98920880 [sys.catalog]: This master's current role is: LEADER
I20260619 08:59:59.206373 20696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:59:59.206867 20696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:59:59.208261 20696 catalog_manager.cc:1347] Generated new cluster ID: b8118358c7224824a9e822dd95a17ba4
I20260619 08:59:59.208305 20696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 08:59:59 dist-test-slave-h4g5 krb5kdc[20601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 08:59:59.212993 20686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:59.201774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52017 (local address 127.0.0.1:34601)
0619 08:59:59.201945 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:59.201955 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:59.201977 (+ 22us) server_negotiation.cc:407] Connection header received
0619 08:59:59.202691 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:59.202699 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:59.202747 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:59.202856 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:59.203648 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:59.204945 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:59.206003 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:59.206592 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:59.206702 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:59.209520 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:59.209523 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:59.209524 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:59:59.209534 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:59.211321 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:59.211732 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:59.211735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:59.211735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:59.211827 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:59.212035 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:59.212035 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:59.212036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:59.212338 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:59.212481 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:59.212620 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 08:59:59.212708 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":74,"threads_started":1}
I20260619 08:59:59.214112 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 08:59:59.222201 20686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 08:59:59.215518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37527 (local address 127.0.0.1:34601)
0619 08:59:59.215583 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 08:59:59.215584 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 08:59:59.215660 (+ 76us) server_negotiation.cc:407] Connection header received
0619 08:59:59.215736 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 08:59:59.215736 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 08:59:59.215744 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 08:59:59.215770 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0619 08:59:59.216402 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:59.217123 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:59.218044 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 08:59:59.218362 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 08:59:59.218394 (+ 32us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 08:59:59.219716 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 08:59:59.219717 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 08:59:59.219717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 08:59:59.219723 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 08:59:59.221092 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:59.221456 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:59.221457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:59.221458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:59.221502 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 08:59:59.221679 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 08:59:59.221679 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 08:59:59.221679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 08:59:59.221783 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 08:59:59.221850 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 08:59:59.222043 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 08:59:59.222100 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 08:59:59.228365 20696 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:59:59.228843 20696 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:59:59.234793 20696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32e1c6d700ed4cd5b38cc78c98920880: Generated new TSK 0
I20260619 08:59:59.234973 20696 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:59:59.239027 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 20614
W20260619 08:59:59.242893 20599 connection.cc:535] client connection to 127.0.0.1:34601 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34601): Connection reset by peer (error 104)
2026-06-19T08:59:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1205 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 08:59:59 dist-test-slave-h4g5 krb5kdc[20713](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 08:59:59 dist-test-slave-h4g5 krb5kdc[20713](info): set up 2 sockets
Jun 19 08:59:59 dist-test-slave-h4g5 krb5kdc[20713](info): commencing operation
krb5kdc: starting...
W20260619 09:00:00.207878 392 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 09:00:00 dist-test-slave-h4g5 krb5kdc[20713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:00:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:00:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 09:00:00.319269 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37241
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:00:00.400135 20726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:00:00.400349 20726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:00:00.400380 20726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:00:00.401542 20726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:00:00.401584 20726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:00:00.401602 20726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:00:00.401618 20726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:00:00.401634 20726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:00:00.403139 20726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37241
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:00:00.403388 20726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:00:00.403594 20726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:00:00.408541 20734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:00.408591 20732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:00.408558 20731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:00:00.408878 20726 server_base.cc:1034] running on GCE node
I20260619 09:00:00.409124 20726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:00:00.409513 20726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:00:00.410691 20726 hybrid_clock.cc:648] HybridClock initialized: now 1781859600410668 us; error 28 us; skew 500 ppm
Jun 19 09:00:00 dist-test-slave-h4g5 krb5kdc[20713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:00:00.417546 20726 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 09:00:00.418012 20726 webserver.cc:460] Webserver started at http://127.0.0.1:39957/ using document root <none> and password file <none>
I20260619 09:00:00.418243 20726 fs_manager.cc:359] Metadata directory not provided
I20260619 09:00:00.418294 20726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:00:00.418419 20726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:00:00.419507 20726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "5a3cfa628bf24fffab8c9d72e73fba5a"
format_stamp: "Formatted at 2026-06-19 09:00:00 on dist-test-slave-h4g5"
server_key: "161c6d3d37c12f1aa31018560dd60b85"
server_key_iv: "20df76573242f189d958848d343acab9"
server_key_version: "encryptionkey@0"
I20260619 09:00:00.419838 20726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "5a3cfa628bf24fffab8c9d72e73fba5a"
format_stamp: "Formatted at 2026-06-19 09:00:00 on dist-test-slave-h4g5"
server_key: "161c6d3d37c12f1aa31018560dd60b85"
server_key_iv: "20df76573242f189d958848d343acab9"
server_key_version: "encryptionkey@0"
I20260619 09:00:00.421209 20726 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:00:00.421933 20741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:00:00.422103 20726 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:00:00.422231 20726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "5a3cfa628bf24fffab8c9d72e73fba5a"
format_stamp: "Formatted at 2026-06-19 09:00:00 on dist-test-slave-h4g5"
server_key: "161c6d3d37c12f1aa31018560dd60b85"
server_key_iv: "20df76573242f189d958848d343acab9"
server_key_version: "encryptionkey@0"
I20260619 09:00:00.422294 20726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:00:00.433547 20726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:00:00.435627 20726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:00:00.435760 20726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:00:00.439532 20726 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37241
I20260619 09:00:00.439574 20793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37241 every 8 connection(s)
I20260619 09:00:00.439891 20726 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:00:00.440724 20794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:00:00.442891 20794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a3cfa628bf24fffab8c9d72e73fba5a: Bootstrap starting.
I20260619 09:00:00.443446 20794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a3cfa628bf24fffab8c9d72e73fba5a: Neither blocks nor log segments found. Creating new log.
I20260619 09:00:00.443693 20794 log.cc:824] T 00000000000000000000000000000000 P 5a3cfa628bf24fffab8c9d72e73fba5a: Log is configured to *not* fsync() on all Append() calls
I20260619 09:00:00.443813 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 20726
I20260619 09:00:00.443938 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 09:00:00.444144 392 external_mini_cluster.cc:1428] Setting key 3c3647171deb0530893a327c27fc21af
I20260619 09:00:00.444403 20794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a3cfa628bf24fffab8c9d72e73fba5a: No bootstrap required, opened a new log
I20260619 09:00:00.446187 20794 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a3cfa628bf24fffab8c9d72e73fba5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3cfa628bf24fffab8c9d72e73fba5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37241 } }
I20260619 09:00:00.446357 20794 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a3cfa628bf24fffab8c9d72e73fba5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:00:00.446380 20794 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a3cfa628bf24fffab8c9d72e73fba5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a3cfa628bf24fffab8c9d72e73fba5a, State: Initialized, Role: FOLLOWER
I20260619 09:00:00.446496 20794 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a3cfa628bf24fffab8c9d72e73fba5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3cfa628bf24fffab8c9d72e73fba5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37241 } }
I20260619 09:00:00.446568 20794 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a3cfa628bf24fffab8c9d72e73fba5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:00:00.446590 20794 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a3cfa628bf24fffab8c9d72e73fba5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:00:00.446635 20794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a3cfa628bf24fffab8c9d72e73fba5a [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:00:00.447166 20794 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a3cfa628bf24fffab8c9d72e73fba5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3cfa628bf24fffab8c9d72e73fba5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37241 } }
I20260619 09:00:00.447256 20794 leader_election.cc:302] T 00000000000000000000000000000000 P 5a3cfa628bf24fffab8c9d72e73fba5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a3cfa628bf24fffab8c9d72e73fba5a; no voters:
I20260619 09:00:00.447418 20794 leader_election.cc:288] T 00000000000000000000000000000000 P 5a3cfa628bf24fffab8c9d72e73fba5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:00:00.447455 20799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a3cfa628bf24fffab8c9d72e73fba5a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:00:00.447556 20799 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a3cfa628bf24fffab8c9d72e73fba5a [term 1 LEADER]: Becoming Leader. State: Replica: 5a3cfa628bf24fffab8c9d72e73fba5a, State: Running, Role: LEADER
I20260619 09:00:00.447671 20794 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a3cfa628bf24fffab8c9d72e73fba5a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:00:00.447682 20799 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a3cfa628bf24fffab8c9d72e73fba5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3cfa628bf24fffab8c9d72e73fba5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37241 } }
I20260619 09:00:00.448071 20799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a3cfa628bf24fffab8c9d72e73fba5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a3cfa628bf24fffab8c9d72e73fba5a. Latest consensus state: current_term: 1 leader_uuid: "5a3cfa628bf24fffab8c9d72e73fba5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3cfa628bf24fffab8c9d72e73fba5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37241 } } }
I20260619 09:00:00.448153 20800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a3cfa628bf24fffab8c9d72e73fba5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a3cfa628bf24fffab8c9d72e73fba5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3cfa628bf24fffab8c9d72e73fba5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37241 } } }
I20260619 09:00:00.448226 20799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a3cfa628bf24fffab8c9d72e73fba5a [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:00.448246 20800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a3cfa628bf24fffab8c9d72e73fba5a [sys.catalog]: This master's current role is: LEADER
W20260619 09:00:00.449484 20815 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5a3cfa628bf24fffab8c9d72e73fba5a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 09:00:00.449539 20815 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 09:00:00.449695 20808 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:00:00.449931 20808 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:00:00.450558 20798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:00.444657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56651 (local address 127.0.0.1:37241)
0619 09:00:00.444846 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:00.444855 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:00.444884 (+ 29us) server_negotiation.cc:407] Connection header received
0619 09:00:00.445553 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:00.445561 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:00.445608 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:00.445780 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:00.445980 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:00.447223 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:00.448244 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:00.448846 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:00.448931 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:00.449862 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:00.449866 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:00.449866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:00:00.450058 (+ 192us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:00.450096 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:00:00.450109 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:00.450191 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:00.450314 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 09:00:00.450341 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260619 09:00:00.451396 20808 catalog_manager.cc:1347] Generated new cluster ID: 7181c99c2dad456ba31d862793d62809
I20260619 09:00:00.451541 20808 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:00:00.451531 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:00:00.456311 20798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:00.453033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37223 (local address 127.0.0.1:37241)
0619 09:00:00.453066 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:00.453067 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:00.453132 (+ 65us) server_negotiation.cc:407] Connection header received
0619 09:00:00.453199 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:00.453200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:00.453209 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:00.453275 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:00.453393 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:00.454121 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:00.455064 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:00.455378 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:00.455446 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:00.456053 (+ 607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:00.456054 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:00.456055 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:00:00.456059 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:00.456077 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:00:00.456104 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:00.456165 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:00.456227 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 09:00:00.456238 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 09:00:00.461374 20808 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:00:00.461879 20808 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:00:00.468659 20808 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a3cfa628bf24fffab8c9d72e73fba5a: Generated new TSK 0
I20260619 09:00:00.468827 20808 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:00:00.474900 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 20726
W20260619 09:00:00.478678 20711 connection.cc:535] client connection to 127.0.0.1:37241 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37241): Connection reset by peer (error 104)
2026-06-19T09:00:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1235 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:00:00 dist-test-slave-h4g5 krb5kdc[20825](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:00:00 dist-test-slave-h4g5 krb5kdc[20825](info): set up 2 sockets
Jun 19 09:00:00 dist-test-slave-h4g5 krb5kdc[20825](info): commencing operation
krb5kdc: starting...
W20260619 09:00:01.427838 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 09:00:01 dist-test-slave-h4g5 krb5kdc[20825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:00:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:00:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 09:00:01.541100 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46725
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:00:01.621663 20838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:00:01.621888 20838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:00:01.621918 20838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:00:01.623090 20838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:00:01.623136 20838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:00:01.623154 20838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:00:01.623169 20838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:00:01.623184 20838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:00:01.624744 20838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46725
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:00:01.624997 20838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:00:01.625201 20838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:00:01.629823 20844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:01.629840 20846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:01.629801 20843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:00:01.629936 20838 server_base.cc:1034] running on GCE node
I20260619 09:00:01.630177 20838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:00:01.630440 20838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:00:01.631572 20838 hybrid_clock.cc:648] HybridClock initialized: now 1781859601631557 us; error 29 us; skew 500 ppm
Jun 19 09:00:01 dist-test-slave-h4g5 krb5kdc[20825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:00:01.633947 20838 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 09:00:01.634366 20838 webserver.cc:460] Webserver started at http://127.0.0.1:41037/ using document root <none> and password file <none>
I20260619 09:00:01.634541 20838 fs_manager.cc:359] Metadata directory not provided
I20260619 09:00:01.634578 20838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:00:01.634667 20838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:00:01.635658 20838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "02975a70d42c416aaff1e3e38828ceb6"
format_stamp: "Formatted at 2026-06-19 09:00:01 on dist-test-slave-h4g5"
server_key: "7a96f34690aa2e1d4b490629b3eed94e"
server_key_iv: "800e8069e171db09166bdffaee10f5d3"
server_key_version: "encryptionkey@0"
I20260619 09:00:01.635967 20838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "02975a70d42c416aaff1e3e38828ceb6"
format_stamp: "Formatted at 2026-06-19 09:00:01 on dist-test-slave-h4g5"
server_key: "7a96f34690aa2e1d4b490629b3eed94e"
server_key_iv: "800e8069e171db09166bdffaee10f5d3"
server_key_version: "encryptionkey@0"
I20260619 09:00:01.637219 20838 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:00:01.637905 20853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:00:01.638052 20838 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:00:01.638132 20838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "02975a70d42c416aaff1e3e38828ceb6"
format_stamp: "Formatted at 2026-06-19 09:00:01 on dist-test-slave-h4g5"
server_key: "7a96f34690aa2e1d4b490629b3eed94e"
server_key_iv: "800e8069e171db09166bdffaee10f5d3"
server_key_version: "encryptionkey@0"
I20260619 09:00:01.638195 20838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:00:01.641430 20838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:00:01.643179 20838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:00:01.643303 20838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:00:01.647099 20838 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46725
I20260619 09:00:01.647126 20905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46725 every 8 connection(s)
I20260619 09:00:01.647435 20838 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:00:01.648425 20906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:00:01.650640 20906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02975a70d42c416aaff1e3e38828ceb6: Bootstrap starting.
I20260619 09:00:01.651144 20906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02975a70d42c416aaff1e3e38828ceb6: Neither blocks nor log segments found. Creating new log.
I20260619 09:00:01.651369 20906 log.cc:824] T 00000000000000000000000000000000 P 02975a70d42c416aaff1e3e38828ceb6: Log is configured to *not* fsync() on all Append() calls
I20260619 09:00:01.652179 20906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02975a70d42c416aaff1e3e38828ceb6: No bootstrap required, opened a new log
I20260619 09:00:01.653735 20906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02975a70d42c416aaff1e3e38828ceb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02975a70d42c416aaff1e3e38828ceb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46725 } }
I20260619 09:00:01.653867 20906 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02975a70d42c416aaff1e3e38828ceb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:00:01.653888 20906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02975a70d42c416aaff1e3e38828ceb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02975a70d42c416aaff1e3e38828ceb6, State: Initialized, Role: FOLLOWER
I20260619 09:00:01.653990 20906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02975a70d42c416aaff1e3e38828ceb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02975a70d42c416aaff1e3e38828ceb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46725 } }
I20260619 09:00:01.654054 20906 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02975a70d42c416aaff1e3e38828ceb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:00:01.654078 20906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02975a70d42c416aaff1e3e38828ceb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:00:01.654109 20906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02975a70d42c416aaff1e3e38828ceb6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:00:01.654618 20906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02975a70d42c416aaff1e3e38828ceb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02975a70d42c416aaff1e3e38828ceb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46725 } }
I20260619 09:00:01.654698 20906 leader_election.cc:302] T 00000000000000000000000000000000 P 02975a70d42c416aaff1e3e38828ceb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02975a70d42c416aaff1e3e38828ceb6; no voters:
I20260619 09:00:01.654824 20906 leader_election.cc:288] T 00000000000000000000000000000000 P 02975a70d42c416aaff1e3e38828ceb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:00:01.654881 20909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02975a70d42c416aaff1e3e38828ceb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:00:01.655064 20906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02975a70d42c416aaff1e3e38828ceb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:00:01.655236 20909 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02975a70d42c416aaff1e3e38828ceb6 [term 1 LEADER]: Becoming Leader. State: Replica: 02975a70d42c416aaff1e3e38828ceb6, State: Running, Role: LEADER
I20260619 09:00:01.655396 20909 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02975a70d42c416aaff1e3e38828ceb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02975a70d42c416aaff1e3e38828ceb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46725 } }
I20260619 09:00:01.655660 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 20838
I20260619 09:00:01.655772 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 09:00:01.655790 20910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02975a70d42c416aaff1e3e38828ceb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02975a70d42c416aaff1e3e38828ceb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02975a70d42c416aaff1e3e38828ceb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46725 } } }
I20260619 09:00:01.655802 20911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02975a70d42c416aaff1e3e38828ceb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02975a70d42c416aaff1e3e38828ceb6. Latest consensus state: current_term: 1 leader_uuid: "02975a70d42c416aaff1e3e38828ceb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02975a70d42c416aaff1e3e38828ceb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46725 } } }
I20260619 09:00:01.655870 20910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02975a70d42c416aaff1e3e38828ceb6 [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:01.655886 20911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02975a70d42c416aaff1e3e38828ceb6 [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:01.655951 392 external_mini_cluster.cc:1428] Setting key 50bcd96cba80043761632c0399c4f364
I20260619 09:00:01.656076 20918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:00:01.656838 20918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:00:01.658233 20927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:01.656502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50017 (local address 127.0.0.1:46725)
0619 09:00:01.656721 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:01.656731 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:01.656764 (+ 33us) server_negotiation.cc:407] Connection header received
0619 09:00:01.656880 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:01.656887 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:01.656922 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:01.657108 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:01.657734 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:01.657737 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:01.657738 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:00:01.657741 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:01.657780 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:00:01.657794 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:01.657847 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:01.658007 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 09:00:01.658051 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":76,"threads_started":1}
I20260619 09:00:01.658515 20918 catalog_manager.cc:1347] Generated new cluster ID: 2da66c2d87b7436f92e59013156cf754
I20260619 09:00:01.658560 20918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:00:01.659052 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:00:01.661585 20927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:01.660495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51709 (local address 127.0.0.1:46725)
0619 09:00:01.660524 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:01.660525 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:01.660645 (+ 120us) server_negotiation.cc:407] Connection header received
0619 09:00:01.660724 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:01.660725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:01.660739 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:01.660763 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:01.661378 (+ 615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:01.661379 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:01.661380 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:00:01.661383 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:01.661396 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:00:01.661401 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:01.661428 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:01.661515 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 09:00:01.661527 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260619 09:00:01.672758 20918 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:00:01.672966 20918 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:00:01.681274 20918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02975a70d42c416aaff1e3e38828ceb6: Generated new TSK 0
I20260619 09:00:01.681478 20918 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:00:01.686887 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 20838
2026-06-19T09:00:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1212 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:00:01 dist-test-slave-h4g5 krb5kdc[20937](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:00:01 dist-test-slave-h4g5 krb5kdc[20937](info): set up 2 sockets
Jun 19 09:00:01 dist-test-slave-h4g5 krb5kdc[20937](info): commencing operation
krb5kdc: starting...
W20260619 09:00:02.636216 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 09:00:02 dist-test-slave-h4g5 krb5kdc[20937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:00:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:00:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 09:00:02.750814 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44251
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:00:02.830359 20950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:00:02.830588 20950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:00:02.830621 20950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:00:02.831804 20950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:00:02.831849 20950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:00:02.831867 20950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:00:02.831883 20950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:00:02.831897 20950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:00:02.833457 20950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:00:02.833709 20950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:00:02.833916 20950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:00:02.838855 20958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:02.839031 20955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:02.839027 20956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:00:02.839066 20950 server_base.cc:1034] running on GCE node
I20260619 09:00:02.839412 20950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:00:02.839706 20950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:00:02.840871 20950 hybrid_clock.cc:648] HybridClock initialized: now 1781859602840840 us; error 37 us; skew 500 ppm
Jun 19 09:00:02 dist-test-slave-h4g5 krb5kdc[20937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:00:02.843468 20950 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 09:00:02.843937 20950 webserver.cc:460] Webserver started at http://127.0.0.1:38755/ using document root <none> and password file <none>
I20260619 09:00:02.844192 20950 fs_manager.cc:359] Metadata directory not provided
I20260619 09:00:02.844244 20950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:00:02.844365 20950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:00:02.845436 20950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "4a13b0924b5b4ae9803c045736d30766"
format_stamp: "Formatted at 2026-06-19 09:00:02 on dist-test-slave-h4g5"
server_key: "0a61de311fd13635d3c0da62381f7ff2"
server_key_iv: "5a5ce7b45a616c79c47c3099036d82ad"
server_key_version: "encryptionkey@0"
I20260619 09:00:02.845777 20950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "4a13b0924b5b4ae9803c045736d30766"
format_stamp: "Formatted at 2026-06-19 09:00:02 on dist-test-slave-h4g5"
server_key: "0a61de311fd13635d3c0da62381f7ff2"
server_key_iv: "5a5ce7b45a616c79c47c3099036d82ad"
server_key_version: "encryptionkey@0"
I20260619 09:00:02.847038 20950 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 09:00:02.847723 20965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:00:02.847867 20950 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:00:02.847957 20950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "4a13b0924b5b4ae9803c045736d30766"
format_stamp: "Formatted at 2026-06-19 09:00:02 on dist-test-slave-h4g5"
server_key: "0a61de311fd13635d3c0da62381f7ff2"
server_key_iv: "5a5ce7b45a616c79c47c3099036d82ad"
server_key_version: "encryptionkey@0"
I20260619 09:00:02.848014 20950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:00:02.859004 20950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:00:02.861055 20950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:00:02.861191 20950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:00:02.864974 20950 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44251
I20260619 09:00:02.865012 21017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44251 every 8 connection(s)
I20260619 09:00:02.865331 20950 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:00:02.865523 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 20950
I20260619 09:00:02.865644 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 09:00:02.865851 392 external_mini_cluster.cc:1428] Setting key 204bf41b35fb1c1ff9eaf048123555d8
I20260619 09:00:02.866343 21018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:00:02.869433 21018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a13b0924b5b4ae9803c045736d30766: Bootstrap starting.
I20260619 09:00:02.870023 21018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a13b0924b5b4ae9803c045736d30766: Neither blocks nor log segments found. Creating new log.
I20260619 09:00:02.870321 21018 log.cc:824] T 00000000000000000000000000000000 P 4a13b0924b5b4ae9803c045736d30766: Log is configured to *not* fsync() on all Append() calls
I20260619 09:00:02.871028 21018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a13b0924b5b4ae9803c045736d30766: No bootstrap required, opened a new log
Jun 19 09:00:02 dist-test-slave-h4g5 krb5kdc[20937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 09:00:02.872907 21018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a13b0924b5b4ae9803c045736d30766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a13b0924b5b4ae9803c045736d30766" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44251 } }
I20260619 09:00:02.873095 21018 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a13b0924b5b4ae9803c045736d30766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:00:02.873134 21018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a13b0924b5b4ae9803c045736d30766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a13b0924b5b4ae9803c045736d30766, State: Initialized, Role: FOLLOWER
I20260619 09:00:02.873282 21018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a13b0924b5b4ae9803c045736d30766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a13b0924b5b4ae9803c045736d30766" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44251 } }
I20260619 09:00:02.873356 21018 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a13b0924b5b4ae9803c045736d30766 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:00:02.873390 21018 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a13b0924b5b4ae9803c045736d30766 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:00:02.873463 21018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a13b0924b5b4ae9803c045736d30766 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:00:02.874015 21018 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a13b0924b5b4ae9803c045736d30766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a13b0924b5b4ae9803c045736d30766" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44251 } }
I20260619 09:00:02.874102 21018 leader_election.cc:302] T 00000000000000000000000000000000 P 4a13b0924b5b4ae9803c045736d30766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a13b0924b5b4ae9803c045736d30766; no voters:
I20260619 09:00:02.874281 21018 leader_election.cc:288] T 00000000000000000000000000000000 P 4a13b0924b5b4ae9803c045736d30766 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:00:02.874323 21023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a13b0924b5b4ae9803c045736d30766 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:00:02.874436 21023 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a13b0924b5b4ae9803c045736d30766 [term 1 LEADER]: Becoming Leader. State: Replica: 4a13b0924b5b4ae9803c045736d30766, State: Running, Role: LEADER
I20260619 09:00:02.874558 21018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a13b0924b5b4ae9803c045736d30766 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:00:02.874578 21023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a13b0924b5b4ae9803c045736d30766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a13b0924b5b4ae9803c045736d30766" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44251 } }
I20260619 09:00:02.874943 21023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a13b0924b5b4ae9803c045736d30766 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a13b0924b5b4ae9803c045736d30766. Latest consensus state: current_term: 1 leader_uuid: "4a13b0924b5b4ae9803c045736d30766" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a13b0924b5b4ae9803c045736d30766" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44251 } } }
I20260619 09:00:02.875068 21023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a13b0924b5b4ae9803c045736d30766 [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:02.875312 21030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:00:02.875787 21024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a13b0924b5b4ae9803c045736d30766 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a13b0924b5b4ae9803c045736d30766" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a13b0924b5b4ae9803c045736d30766" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44251 } } }
I20260619 09:00:02.875854 21024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a13b0924b5b4ae9803c045736d30766 [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:02.876127 21030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:00:02.877087 21021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:02.866326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58203 (local address 127.0.0.1:44251)
0619 09:00:02.866536 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:02.866545 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:02.866574 (+ 29us) server_negotiation.cc:407] Connection header received
0619 09:00:02.867062 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:02.867072 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:02.867145 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:02.867294 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:02.867873 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:02.869240 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:02.870019 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:02.870602 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:02.870683 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:02.872843 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:02.872847 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:02.872847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:02.872859 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:02.874831 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:02.875260 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:02.875264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:02.875264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:02.875362 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:02.875576 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:02.875576 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:02.875577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:02.875877 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:02.876767 (+ 890us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:02.876825 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 09:00:02.876930 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":70,"threads_started":1}
I20260619 09:00:02.877863 21030 catalog_manager.cc:1347] Generated new cluster ID: 860b873586a144949c41cc7d483bd0e6
I20260619 09:00:02.877913 21030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:00:02.878001 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:00:02.884992 21021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:02.879207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43161 (local address 127.0.0.1:44251)
0619 09:00:02.879245 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:02.879246 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:02.879342 (+ 96us) server_negotiation.cc:407] Connection header received
0619 09:00:02.879393 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:02.879394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:02.879403 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:02.879446 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:02.879926 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:02.880518 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:02.881197 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:02.881421 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:02.881473 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:02.882673 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:02.882675 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:02.882675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:02.882682 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:02.883926 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:02.884293 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:02.884294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:02.884295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:02.884351 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:02.884517 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:02.884518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:02.884518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:02.884637 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:02.884729 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:02.884823 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 09:00:02.884899 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":11,"server-negotiator.queue_time_us":20}
I20260619 09:00:02.886739 21030 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:00:02.887449 21030 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:00:02.896127 21030 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a13b0924b5b4ae9803c045736d30766: Generated new TSK 0
I20260619 09:00:02.896363 21030 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:00:02.903959 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 20950
2026-06-19T09:00:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1217 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:00:02 dist-test-slave-h4g5 krb5kdc[21049](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:00:02 dist-test-slave-h4g5 krb5kdc[21049](info): set up 2 sockets
Jun 19 09:00:02 dist-test-slave-h4g5 krb5kdc[21049](info): commencing operation
krb5kdc: starting...
W20260619 09:00:03.860049 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 09:00:03 dist-test-slave-h4g5 krb5kdc[21049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:00:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:00:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 09:00:03.970227 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45739
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:00:04.049397 21062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:00:04.049605 21062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:00:04.049636 21062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:00:04.050786 21062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:00:04.050825 21062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:00:04.050843 21062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:00:04.050858 21062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:00:04.050874 21062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:00:04.052410 21062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:00:04.052642 21062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:00:04.052842 21062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 09:00:04.057538 21062 server_base.cc:1034] running on GCE node
W20260619 09:00:04.057530 21067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:04.057650 21070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:04.057650 21068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:00:04.058058 21062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:00:04.058326 21062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:00:04.059494 21062 hybrid_clock.cc:648] HybridClock initialized: now 1781859604059481 us; error 29 us; skew 500 ppm
Jun 19 09:00:04 dist-test-slave-h4g5 krb5kdc[21049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:00:04.061980 21062 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 09:00:04.062449 21062 webserver.cc:460] Webserver started at http://127.0.0.1:33861/ using document root <none> and password file <none>
I20260619 09:00:04.062672 21062 fs_manager.cc:359] Metadata directory not provided
I20260619 09:00:04.062729 21062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:00:04.062853 21062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:00:04.063899 21062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "84acdc076a104741ba191cac962626f8"
format_stamp: "Formatted at 2026-06-19 09:00:04 on dist-test-slave-h4g5"
server_key: "9995f20d4d1704bacf40a47bf4d3e347"
server_key_iv: "b4efea7b102017b4926f1c11c7624e12"
server_key_version: "encryptionkey@0"
I20260619 09:00:04.064254 21062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "84acdc076a104741ba191cac962626f8"
format_stamp: "Formatted at 2026-06-19 09:00:04 on dist-test-slave-h4g5"
server_key: "9995f20d4d1704bacf40a47bf4d3e347"
server_key_iv: "b4efea7b102017b4926f1c11c7624e12"
server_key_version: "encryptionkey@0"
I20260619 09:00:04.065485 21062 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 09:00:04.066215 21077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:00:04.066365 21062 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:00:04.066439 21062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "84acdc076a104741ba191cac962626f8"
format_stamp: "Formatted at 2026-06-19 09:00:04 on dist-test-slave-h4g5"
server_key: "9995f20d4d1704bacf40a47bf4d3e347"
server_key_iv: "b4efea7b102017b4926f1c11c7624e12"
server_key_version: "encryptionkey@0"
I20260619 09:00:04.066500 21062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:00:04.082643 21062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:00:04.084832 21062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:00:04.084971 21062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:00:04.088635 21062 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45739
I20260619 09:00:04.088678 21129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45739 every 8 connection(s)
I20260619 09:00:04.089011 21062 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:00:04.089856 21130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:00:04.092175 21130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84acdc076a104741ba191cac962626f8: Bootstrap starting.
I20260619 09:00:04.092701 21130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84acdc076a104741ba191cac962626f8: Neither blocks nor log segments found. Creating new log.
I20260619 09:00:04.092954 21130 log.cc:824] T 00000000000000000000000000000000 P 84acdc076a104741ba191cac962626f8: Log is configured to *not* fsync() on all Append() calls
I20260619 09:00:04.093587 21130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84acdc076a104741ba191cac962626f8: No bootstrap required, opened a new log
I20260619 09:00:04.094933 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 21062
I20260619 09:00:04.095039 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 09:00:04.095219 392 external_mini_cluster.cc:1428] Setting key b3bfd827673d2e90e56a8e51def9c96d
I20260619 09:00:04.095440 21130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84acdc076a104741ba191cac962626f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84acdc076a104741ba191cac962626f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45739 } }
I20260619 09:00:04.095573 21130 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84acdc076a104741ba191cac962626f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:00:04.095603 21130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84acdc076a104741ba191cac962626f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84acdc076a104741ba191cac962626f8, State: Initialized, Role: FOLLOWER
I20260619 09:00:04.095757 21130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84acdc076a104741ba191cac962626f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84acdc076a104741ba191cac962626f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45739 } }
I20260619 09:00:04.095834 21130 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84acdc076a104741ba191cac962626f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:00:04.095871 21130 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84acdc076a104741ba191cac962626f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:00:04.095911 21130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84acdc076a104741ba191cac962626f8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:00:04.096969 21130 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84acdc076a104741ba191cac962626f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84acdc076a104741ba191cac962626f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45739 } }
I20260619 09:00:04.097076 21130 leader_election.cc:302] T 00000000000000000000000000000000 P 84acdc076a104741ba191cac962626f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84acdc076a104741ba191cac962626f8; no voters:
I20260619 09:00:04.097229 21130 leader_election.cc:288] T 00000000000000000000000000000000 P 84acdc076a104741ba191cac962626f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:00:04.097259 21135 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84acdc076a104741ba191cac962626f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:00:04.097419 21135 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84acdc076a104741ba191cac962626f8 [term 1 LEADER]: Becoming Leader. State: Replica: 84acdc076a104741ba191cac962626f8, State: Running, Role: LEADER
I20260619 09:00:04.097469 21130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84acdc076a104741ba191cac962626f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:00:04.097507 21135 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84acdc076a104741ba191cac962626f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84acdc076a104741ba191cac962626f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45739 } }
I20260619 09:00:04.098080 21137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84acdc076a104741ba191cac962626f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84acdc076a104741ba191cac962626f8. Latest consensus state: current_term: 1 leader_uuid: "84acdc076a104741ba191cac962626f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84acdc076a104741ba191cac962626f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45739 } } }
I20260619 09:00:04.098186 21137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84acdc076a104741ba191cac962626f8 [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:04.098383 21139 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:00:04.098891 21139 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:00:04.098083 21136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84acdc076a104741ba191cac962626f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84acdc076a104741ba191cac962626f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84acdc076a104741ba191cac962626f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45739 } } }
I20260619 09:00:04.099241 21136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84acdc076a104741ba191cac962626f8 [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:04.101338 21134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:04.095731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46355 (local address 127.0.0.1:45739)
0619 09:00:04.095913 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:04.095923 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:04.096022 (+ 99us) server_negotiation.cc:407] Connection header received
0619 09:00:04.096219 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:04.096228 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:04.096300 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:04.096461 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:04.096607 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:04.098047 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:04.099097 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:04.099873 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:04.099983 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:04.100708 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:04.100712 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:04.100712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:00:04.100882 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:04.100912 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:00:04.100923 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:04.100974 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:04.101068 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 09:00:04.101116 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
I20260619 09:00:04.102155 21139 catalog_manager.cc:1347] Generated new cluster ID: 2498cb31b4604c5cb8141c8d8eaf7c29
I20260619 09:00:04.102203 21139 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:00:04.102317 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:00:04.107472 21134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:04.103616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60315 (local address 127.0.0.1:45739)
0619 09:00:04.103646 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:04.103647 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:04.104266 (+ 619us) server_negotiation.cc:407] Connection header received
0619 09:00:04.104329 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:04.104330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:04.104340 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:04.104387 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:04.104542 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:04.105263 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:04.106245 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:04.106516 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:04.106566 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:04.107239 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:04.107240 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:04.107240 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:00:04.107245 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:04.107260 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:00:04.107264 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:04.107317 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:04.107375 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 09:00:04.107397 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260619 09:00:04.108461 21139 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:00:04.108969 21139 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:00:04.115833 21139 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84acdc076a104741ba191cac962626f8: Generated new TSK 0
I20260619 09:00:04.116024 21139 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:00:04.118745 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 21062
2026-06-19T09:00:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1214 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:00:04 dist-test-slave-h4g5 krb5kdc[21161](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:00:04 dist-test-slave-h4g5 krb5kdc[21161](info): set up 2 sockets
Jun 19 09:00:04 dist-test-slave-h4g5 krb5kdc[21161](info): commencing operation
krb5kdc: starting...
W20260619 09:00:05.080000 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 09:00:05 dist-test-slave-h4g5 krb5kdc[21161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:00:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:00:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 09:00:05.194610 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38997
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:00:05.275704 21174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:00:05.275936 21174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:00:05.275967 21174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:00:05.277158 21174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:00:05.277202 21174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:00:05.277221 21174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:00:05.277237 21174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:00:05.277253 21174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:00:05.278762 21174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:00:05.279022 21174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:00:05.279235 21174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:00:05.283856 21182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:05.283862 21179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:05.283870 21180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:00:05.284158 21174 server_base.cc:1034] running on GCE node
I20260619 09:00:05.284359 21174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:00:05.284600 21174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:00:05.285797 21174 hybrid_clock.cc:648] HybridClock initialized: now 1781859605285771 us; error 41 us; skew 500 ppm
Jun 19 09:00:05 dist-test-slave-h4g5 krb5kdc[21161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:00:05.288326 21174 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 09:00:05.288744 21174 webserver.cc:460] Webserver started at http://127.0.0.1:37633/ using document root <none> and password file <none>
I20260619 09:00:05.288975 21174 fs_manager.cc:359] Metadata directory not provided
I20260619 09:00:05.289021 21174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:00:05.289132 21174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:00:05.290220 21174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "bbd72261213c471cb3247a2fa2e1fd95"
format_stamp: "Formatted at 2026-06-19 09:00:05 on dist-test-slave-h4g5"
server_key: "48a2c778ecdfb3179d059e69736f304a"
server_key_iv: "74869a398c41cee296e0a94df6248d0a"
server_key_version: "encryptionkey@0"
I20260619 09:00:05.290540 21174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "bbd72261213c471cb3247a2fa2e1fd95"
format_stamp: "Formatted at 2026-06-19 09:00:05 on dist-test-slave-h4g5"
server_key: "48a2c778ecdfb3179d059e69736f304a"
server_key_iv: "74869a398c41cee296e0a94df6248d0a"
server_key_version: "encryptionkey@0"
I20260619 09:00:05.291898 21174 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:00:05.292723 21189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:00:05.292896 21174 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 09:00:05.292995 21174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "bbd72261213c471cb3247a2fa2e1fd95"
format_stamp: "Formatted at 2026-06-19 09:00:05 on dist-test-slave-h4g5"
server_key: "48a2c778ecdfb3179d059e69736f304a"
server_key_iv: "74869a398c41cee296e0a94df6248d0a"
server_key_version: "encryptionkey@0"
I20260619 09:00:05.293056 21174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:00:05.296330 21174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:00:05.298245 21174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:00:05.298350 21174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:00:05.301972 21174 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38997
I20260619 09:00:05.302023 21241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38997 every 8 connection(s)
I20260619 09:00:05.302311 21174 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:00:05.303268 21242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:00:05.305646 21242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbd72261213c471cb3247a2fa2e1fd95: Bootstrap starting.
I20260619 09:00:05.306217 21242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbd72261213c471cb3247a2fa2e1fd95: Neither blocks nor log segments found. Creating new log.
I20260619 09:00:05.306468 21242 log.cc:824] T 00000000000000000000000000000000 P bbd72261213c471cb3247a2fa2e1fd95: Log is configured to *not* fsync() on all Append() calls
I20260619 09:00:05.307119 21242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbd72261213c471cb3247a2fa2e1fd95: No bootstrap required, opened a new log
I20260619 09:00:05.308753 21242 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbd72261213c471cb3247a2fa2e1fd95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbd72261213c471cb3247a2fa2e1fd95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38997 } }
I20260619 09:00:05.308916 21242 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbd72261213c471cb3247a2fa2e1fd95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:00:05.308962 21242 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbd72261213c471cb3247a2fa2e1fd95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbd72261213c471cb3247a2fa2e1fd95, State: Initialized, Role: FOLLOWER
I20260619 09:00:05.309118 21242 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbd72261213c471cb3247a2fa2e1fd95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbd72261213c471cb3247a2fa2e1fd95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38997 } }
I20260619 09:00:05.309197 21242 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbd72261213c471cb3247a2fa2e1fd95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:00:05.309228 21242 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbd72261213c471cb3247a2fa2e1fd95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:00:05.309276 21242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbd72261213c471cb3247a2fa2e1fd95 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:00:05.309518 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 21174
I20260619 09:00:05.309607 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 09:00:05.309764 392 external_mini_cluster.cc:1428] Setting key 6288ed52c6f5993db72fb44359451a60
I20260619 09:00:05.309862 21242 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbd72261213c471cb3247a2fa2e1fd95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbd72261213c471cb3247a2fa2e1fd95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38997 } }
I20260619 09:00:05.309973 21242 leader_election.cc:302] T 00000000000000000000000000000000 P bbd72261213c471cb3247a2fa2e1fd95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbd72261213c471cb3247a2fa2e1fd95; no voters:
I20260619 09:00:05.310146 21242 leader_election.cc:288] T 00000000000000000000000000000000 P bbd72261213c471cb3247a2fa2e1fd95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:00:05.310384 21245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbd72261213c471cb3247a2fa2e1fd95 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:00:05.310402 21242 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbd72261213c471cb3247a2fa2e1fd95 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:00:05.310606 21245 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbd72261213c471cb3247a2fa2e1fd95 [term 1 LEADER]: Becoming Leader. State: Replica: bbd72261213c471cb3247a2fa2e1fd95, State: Running, Role: LEADER
I20260619 09:00:05.310730 21245 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbd72261213c471cb3247a2fa2e1fd95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbd72261213c471cb3247a2fa2e1fd95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38997 } }
I20260619 09:00:05.311118 21245 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbd72261213c471cb3247a2fa2e1fd95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbd72261213c471cb3247a2fa2e1fd95. Latest consensus state: current_term: 1 leader_uuid: "bbd72261213c471cb3247a2fa2e1fd95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbd72261213c471cb3247a2fa2e1fd95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38997 } } }
I20260619 09:00:05.311236 21245 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbd72261213c471cb3247a2fa2e1fd95 [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:05.311645 21254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:00:05.312137 21248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:05.310221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42631 (local address 127.0.0.1:38997)
0619 09:00:05.310378 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:05.310388 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:05.310424 (+ 36us) server_negotiation.cc:407] Connection header received
0619 09:00:05.310547 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:05.310555 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:05.310591 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:05.310769 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:05.311530 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:05.311535 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:05.311536 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:00:05.311540 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:05.311589 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:00:05.311603 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:05.311658 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:05.311840 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 09:00:05.311890 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":70,"threads_started":1}
I20260619 09:00:05.312162 21254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:00:05.312763 21247 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbd72261213c471cb3247a2fa2e1fd95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbd72261213c471cb3247a2fa2e1fd95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbd72261213c471cb3247a2fa2e1fd95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38997 } } }
I20260619 09:00:05.312870 21247 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbd72261213c471cb3247a2fa2e1fd95 [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:05.313026 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:00:05.314510 21254 catalog_manager.cc:1347] Generated new cluster ID: 0581610328a94a8db39d2508edb28b98
I20260619 09:00:05.314569 21254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:00:05.315711 21248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:05.314673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55825 (local address 127.0.0.1:38997)
0619 09:00:05.314709 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:05.314710 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:05.314854 (+ 144us) server_negotiation.cc:407] Connection header received
0619 09:00:05.314930 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:05.314931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:05.314936 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:05.314979 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:05.315519 (+ 540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:05.315520 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:05.315520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:00:05.315524 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:05.315538 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:00:05.315542 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:05.315586 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:05.315636 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 09:00:05.315649 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 09:00:05.330806 21254 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:00:05.331027 21254 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:00:05.338917 21254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbd72261213c471cb3247a2fa2e1fd95: Generated new TSK 0
I20260619 09:00:05.339128 21254 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:00:05.343976 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 21174
2026-06-19T09:00:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1225 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:00:05 dist-test-slave-h4g5 krb5kdc[21273](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:00:05 dist-test-slave-h4g5 krb5kdc[21273](info): set up 2 sockets
Jun 19 09:00:05 dist-test-slave-h4g5 krb5kdc[21273](info): commencing operation
krb5kdc: starting...
W20260619 09:00:06.295913 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 09:00:06 dist-test-slave-h4g5 krb5kdc[21273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:00:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:00:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 09:00:06.406515 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43791
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:00:06.484704 21286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:00:06.484917 21286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:00:06.484941 21286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:00:06.486028 21286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:00:06.486060 21286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:00:06.486074 21286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:00:06.486085 21286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:00:06.486099 21286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:00:06.487498 21286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:00:06.487692 21286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:00:06.487851 21286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:00:06.492365 21292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:06.492448 21294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:06.492381 21291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:00:06.492587 21286 server_base.cc:1034] running on GCE node
I20260619 09:00:06.492805 21286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:00:06.493063 21286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:00:06.494225 21286 hybrid_clock.cc:648] HybridClock initialized: now 1781859606494214 us; error 32 us; skew 500 ppm
Jun 19 09:00:06 dist-test-slave-h4g5 krb5kdc[21273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:00:06.499398 21286 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 09:00:06.499840 21286 webserver.cc:460] Webserver started at http://127.0.0.1:46269/ using document root <none> and password file <none>
I20260619 09:00:06.500042 21286 fs_manager.cc:359] Metadata directory not provided
I20260619 09:00:06.500109 21286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:00:06.500219 21286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:00:06.501225 21286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "1a9e328b5ccb4a9ba9fc441799b9c4fd"
format_stamp: "Formatted at 2026-06-19 09:00:06 on dist-test-slave-h4g5"
server_key: "7fbd5392b0af7b9fdb5ba7c497429160"
server_key_iv: "9fd621ea893ab26537974ff17491def4"
server_key_version: "encryptionkey@0"
I20260619 09:00:06.501518 21286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "1a9e328b5ccb4a9ba9fc441799b9c4fd"
format_stamp: "Formatted at 2026-06-19 09:00:06 on dist-test-slave-h4g5"
server_key: "7fbd5392b0af7b9fdb5ba7c497429160"
server_key_iv: "9fd621ea893ab26537974ff17491def4"
server_key_version: "encryptionkey@0"
I20260619 09:00:06.502712 21286 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:00:06.503401 21301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:00:06.503559 21286 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:00:06.503643 21286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "1a9e328b5ccb4a9ba9fc441799b9c4fd"
format_stamp: "Formatted at 2026-06-19 09:00:06 on dist-test-slave-h4g5"
server_key: "7fbd5392b0af7b9fdb5ba7c497429160"
server_key_iv: "9fd621ea893ab26537974ff17491def4"
server_key_version: "encryptionkey@0"
I20260619 09:00:06.503698 21286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:00:06.520959 21286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:00:06.523010 21286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:00:06.523116 21286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:00:06.526686 21286 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43791
I20260619 09:00:06.526729 21353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43791 every 8 connection(s)
I20260619 09:00:06.527024 21286 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:00:06.527840 21354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:00:06.530057 21354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a9e328b5ccb4a9ba9fc441799b9c4fd: Bootstrap starting.
I20260619 09:00:06.530596 21354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a9e328b5ccb4a9ba9fc441799b9c4fd: Neither blocks nor log segments found. Creating new log.
I20260619 09:00:06.530843 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 21286
I20260619 09:00:06.530844 21354 log.cc:824] T 00000000000000000000000000000000 P 1a9e328b5ccb4a9ba9fc441799b9c4fd: Log is configured to *not* fsync() on all Append() calls
I20260619 09:00:06.530930 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 09:00:06.531134 392 external_mini_cluster.cc:1428] Setting key 559779b89a8551b5f1718deebd68bb4a
I20260619 09:00:06.531546 21354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a9e328b5ccb4a9ba9fc441799b9c4fd: No bootstrap required, opened a new log
I20260619 09:00:06.533649 21354 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a9e328b5ccb4a9ba9fc441799b9c4fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a9e328b5ccb4a9ba9fc441799b9c4fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43791 } }
I20260619 09:00:06.533785 21354 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a9e328b5ccb4a9ba9fc441799b9c4fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:00:06.533818 21354 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a9e328b5ccb4a9ba9fc441799b9c4fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9e328b5ccb4a9ba9fc441799b9c4fd, State: Initialized, Role: FOLLOWER
I20260619 09:00:06.533937 21354 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a9e328b5ccb4a9ba9fc441799b9c4fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a9e328b5ccb4a9ba9fc441799b9c4fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43791 } }
I20260619 09:00:06.534013 21354 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a9e328b5ccb4a9ba9fc441799b9c4fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:00:06.534042 21354 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a9e328b5ccb4a9ba9fc441799b9c4fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:00:06.534087 21354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a9e328b5ccb4a9ba9fc441799b9c4fd [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:00:06.534616 21354 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a9e328b5ccb4a9ba9fc441799b9c4fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a9e328b5ccb4a9ba9fc441799b9c4fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43791 } }
I20260619 09:00:06.534708 21354 leader_election.cc:302] T 00000000000000000000000000000000 P 1a9e328b5ccb4a9ba9fc441799b9c4fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a9e328b5ccb4a9ba9fc441799b9c4fd; no voters:
I20260619 09:00:06.534871 21354 leader_election.cc:288] T 00000000000000000000000000000000 P 1a9e328b5ccb4a9ba9fc441799b9c4fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:00:06.534915 21359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a9e328b5ccb4a9ba9fc441799b9c4fd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:00:06.535058 21359 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a9e328b5ccb4a9ba9fc441799b9c4fd [term 1 LEADER]: Becoming Leader. State: Replica: 1a9e328b5ccb4a9ba9fc441799b9c4fd, State: Running, Role: LEADER
I20260619 09:00:06.535136 21354 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a9e328b5ccb4a9ba9fc441799b9c4fd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:00:06.535198 21359 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a9e328b5ccb4a9ba9fc441799b9c4fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a9e328b5ccb4a9ba9fc441799b9c4fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43791 } }
I20260619 09:00:06.535624 21361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a9e328b5ccb4a9ba9fc441799b9c4fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a9e328b5ccb4a9ba9fc441799b9c4fd. Latest consensus state: current_term: 1 leader_uuid: "1a9e328b5ccb4a9ba9fc441799b9c4fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a9e328b5ccb4a9ba9fc441799b9c4fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43791 } } }
I20260619 09:00:06.535702 21361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a9e328b5ccb4a9ba9fc441799b9c4fd [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:06.535629 21360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a9e328b5ccb4a9ba9fc441799b9c4fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a9e328b5ccb4a9ba9fc441799b9c4fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a9e328b5ccb4a9ba9fc441799b9c4fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43791 } } }
I20260619 09:00:06.535923 21360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a9e328b5ccb4a9ba9fc441799b9c4fd [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:06.536317 21368 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:00:06.536864 21368 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:00:06.538282 21368 catalog_manager.cc:1347] Generated new cluster ID: 1a2eb31e4ac34e6fb00657b5a52398db
I20260619 09:00:06.538331 21368 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 09:00:06 dist-test-slave-h4g5 krb5kdc[21273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 09:00:06.542877 21358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:06.531579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:45699 (local address 127.0.0.1:43791)
0619 09:00:06.531716 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:06.531723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:06.531751 (+ 28us) server_negotiation.cc:407] Connection header received
0619 09:00:06.532736 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:06.532744 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:06.532795 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:06.532905 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:06.533643 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:06.534927 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:06.536006 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:06.536678 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:06.536772 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:06.539428 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:06.539431 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:06.539431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:06.539442 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:06.541276 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:06.541695 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:06.541698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:06.541698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:06.541789 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:06.542005 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:06.542005 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:06.542006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:06.542278 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:06.542409 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:06.542591 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 09:00:06.542682 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":65,"threads_started":1}
I20260619 09:00:06.543777 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:00:06.552052 21358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:06.545215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:59219 (local address 127.0.0.1:43791)
0619 09:00:06.545254 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:06.545256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:06.545393 (+ 137us) server_negotiation.cc:407] Connection header received
0619 09:00:06.545453 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:06.545454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:06.545462 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:06.545505 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:06.546086 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:06.546810 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:06.547813 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:06.548152 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:06.548206 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:06.549499 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:06.549500 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:06.549500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:06.549506 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:06.550921 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:06.551303 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:06.551304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:06.551304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:06.551351 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:06.551553 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:06.551554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:06.551554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:06.551664 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:06.551754 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:06.551883 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 09:00:06.551943 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 09:00:06.553143 21368 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:00:06.553606 21368 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:00:06.559182 21368 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a9e328b5ccb4a9ba9fc441799b9c4fd: Generated new TSK 0
I20260619 09:00:06.559357 21368 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:00:06.561806 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 21286
W20260619 09:00:06.565348 21271 connection.cc:535] client connection to 127.0.0.1:43791 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43791): Connection reset by peer (error 104)
2026-06-19T09:00:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1217 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:00:06 dist-test-slave-h4g5 krb5kdc[21385](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:00:06 dist-test-slave-h4g5 krb5kdc[21385](info): set up 2 sockets
Jun 19 09:00:06 dist-test-slave-h4g5 krb5kdc[21385](info): commencing operation
krb5kdc: starting...
W20260619 09:00:07.515980 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 09:00:07 dist-test-slave-h4g5 krb5kdc[21385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:00:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:00:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 09:00:07.628496 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43715
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:00:07.707913 21398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:00:07.708194 21398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:00:07.708225 21398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:00:07.709404 21398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:00:07.709448 21398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:00:07.709461 21398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:00:07.709473 21398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:00:07.709501 21398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:00:07.711021 21398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:00:07.711253 21398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:00:07.711458 21398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:00:07.715947 21404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:07.715947 21403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:07.715999 21406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:00:07.716166 21398 server_base.cc:1034] running on GCE node
I20260619 09:00:07.716454 21398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:00:07.716713 21398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:00:07.717854 21398 hybrid_clock.cc:648] HybridClock initialized: now 1781859607717842 us; error 26 us; skew 500 ppm
Jun 19 09:00:07 dist-test-slave-h4g5 krb5kdc[21385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:00:07.720216 21398 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 09:00:07.720659 21398 webserver.cc:460] Webserver started at http://127.0.0.1:33629/ using document root <none> and password file <none>
I20260619 09:00:07.720870 21398 fs_manager.cc:359] Metadata directory not provided
I20260619 09:00:07.720916 21398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:00:07.721037 21398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:00:07.722059 21398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "9890305420b045a7b296e91cf1407cc3"
format_stamp: "Formatted at 2026-06-19 09:00:07 on dist-test-slave-h4g5"
server_key: "7b5a40302d45eb742413d8e944676578"
server_key_iv: "3577217630827d2c0d2f49d6b7ad7d9a"
server_key_version: "encryptionkey@0"
I20260619 09:00:07.722379 21398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "9890305420b045a7b296e91cf1407cc3"
format_stamp: "Formatted at 2026-06-19 09:00:07 on dist-test-slave-h4g5"
server_key: "7b5a40302d45eb742413d8e944676578"
server_key_iv: "3577217630827d2c0d2f49d6b7ad7d9a"
server_key_version: "encryptionkey@0"
I20260619 09:00:07.723670 21398 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:00:07.724392 21413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:00:07.724539 21398 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:00:07.724619 21398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "9890305420b045a7b296e91cf1407cc3"
format_stamp: "Formatted at 2026-06-19 09:00:07 on dist-test-slave-h4g5"
server_key: "7b5a40302d45eb742413d8e944676578"
server_key_iv: "3577217630827d2c0d2f49d6b7ad7d9a"
server_key_version: "encryptionkey@0"
I20260619 09:00:07.724681 21398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:00:07.750831 21398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:00:07.752892 21398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:00:07.753034 21398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:00:07.756978 21398 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43715
I20260619 09:00:07.757021 21465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43715 every 8 connection(s)
I20260619 09:00:07.757365 21398 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:00:07.758231 21466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:00:07.760589 21466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9890305420b045a7b296e91cf1407cc3: Bootstrap starting.
I20260619 09:00:07.761125 21466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9890305420b045a7b296e91cf1407cc3: Neither blocks nor log segments found. Creating new log.
I20260619 09:00:07.761353 21466 log.cc:824] T 00000000000000000000000000000000 P 9890305420b045a7b296e91cf1407cc3: Log is configured to *not* fsync() on all Append() calls
I20260619 09:00:07.761976 21466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9890305420b045a7b296e91cf1407cc3: No bootstrap required, opened a new log
I20260619 09:00:07.763243 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 21398
I20260619 09:00:07.763343 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 09:00:07.763511 392 external_mini_cluster.cc:1428] Setting key 51706a1a076fc15e0e39f2c36e4d4f52
I20260619 09:00:07.763618 21466 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9890305420b045a7b296e91cf1407cc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9890305420b045a7b296e91cf1407cc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43715 } }
I20260619 09:00:07.763767 21466 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9890305420b045a7b296e91cf1407cc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:00:07.763788 21466 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9890305420b045a7b296e91cf1407cc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9890305420b045a7b296e91cf1407cc3, State: Initialized, Role: FOLLOWER
I20260619 09:00:07.763933 21466 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9890305420b045a7b296e91cf1407cc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9890305420b045a7b296e91cf1407cc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43715 } }
I20260619 09:00:07.764019 21466 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9890305420b045a7b296e91cf1407cc3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:00:07.764048 21466 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9890305420b045a7b296e91cf1407cc3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:00:07.764176 21466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9890305420b045a7b296e91cf1407cc3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:00:07.765134 21466 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9890305420b045a7b296e91cf1407cc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9890305420b045a7b296e91cf1407cc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43715 } }
I20260619 09:00:07.765223 21466 leader_election.cc:302] T 00000000000000000000000000000000 P 9890305420b045a7b296e91cf1407cc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9890305420b045a7b296e91cf1407cc3; no voters:
I20260619 09:00:07.765372 21466 leader_election.cc:288] T 00000000000000000000000000000000 P 9890305420b045a7b296e91cf1407cc3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:00:07.765415 21471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9890305420b045a7b296e91cf1407cc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:00:07.765614 21466 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9890305420b045a7b296e91cf1407cc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:00:07.765623 21471 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9890305420b045a7b296e91cf1407cc3 [term 1 LEADER]: Becoming Leader. State: Replica: 9890305420b045a7b296e91cf1407cc3, State: Running, Role: LEADER
I20260619 09:00:07.765802 21471 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9890305420b045a7b296e91cf1407cc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9890305420b045a7b296e91cf1407cc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43715 } }
I20260619 09:00:07.766173 21472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9890305420b045a7b296e91cf1407cc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9890305420b045a7b296e91cf1407cc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9890305420b045a7b296e91cf1407cc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43715 } } }
I20260619 09:00:07.766261 21472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9890305420b045a7b296e91cf1407cc3 [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:07.766459 21473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9890305420b045a7b296e91cf1407cc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9890305420b045a7b296e91cf1407cc3. Latest consensus state: current_term: 1 leader_uuid: "9890305420b045a7b296e91cf1407cc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9890305420b045a7b296e91cf1407cc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43715 } } }
I20260619 09:00:07.766562 21473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9890305420b045a7b296e91cf1407cc3 [sys.catalog]: This master's current role is: LEADER
W20260619 09:00:07.767269 21487 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9890305420b045a7b296e91cf1407cc3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 09:00:07.767311 21487 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 09:00:07.767361 21480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:00:07.767540 21480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:00:07.769117 21480 catalog_manager.cc:1347] Generated new cluster ID: daf1111828004715aa4a9358a99ca594
I20260619 09:00:07.769136 21470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:07.764032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:44109 (local address 127.0.0.1:43715)
0619 09:00:07.764228 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:07.764239 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:07.764272 (+ 33us) server_negotiation.cc:407] Connection header received
0619 09:00:07.764377 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:07.764383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:07.764428 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:07.764539 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:07.764786 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:07.765922 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:07.766998 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:07.767544 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:07.767654 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:07.768429 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:07.768434 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:07.768434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:00:07.768657 (+ 223us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:07.768695 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:00:07.768709 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:07.768791 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:07.768887 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 09:00:07.768913 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":113,"threads_started":1}
I20260619 09:00:07.769193 21480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:00:07.770007 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:00:07.774748 21470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:07.771421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:37715 (local address 127.0.0.1:43715)
0619 09:00:07.771452 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:07.771454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:07.771549 (+ 95us) server_negotiation.cc:407] Connection header received
0619 09:00:07.771604 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:07.771605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:07.771628 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:07.771680 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:07.771794 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:07.772550 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:07.773509 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:07.773810 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:07.773872 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:07.774502 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:07.774504 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:07.774504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:00:07.774509 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:07.774524 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:00:07.774528 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:07.774585 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:07.774659 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 09:00:07.774674 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 09:00:07.785015 21480 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:00:07.785532 21480 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:00:07.791142 21480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9890305420b045a7b296e91cf1407cc3: Generated new TSK 0
I20260619 09:00:07.791311 21480 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:00:07.796861 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 21398
W20260619 09:00:07.800787 21383 connection.cc:535] client connection to 127.0.0.1:43715 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43715): Connection reset by peer (error 104)
2026-06-19T09:00:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1235 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:00:07 dist-test-slave-h4g5 krb5kdc[21497](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:00:07 dist-test-slave-h4g5 krb5kdc[21497](info): set up 2 sockets
Jun 19 09:00:07 dist-test-slave-h4g5 krb5kdc[21497](info): commencing operation
krb5kdc: starting...
W20260619 09:00:08.747905 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 09:00:08 dist-test-slave-h4g5 krb5kdc[21497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:00:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:00:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 09:00:08.856827 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39585
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:00:08.934206 21510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:00:08.934409 21510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:00:08.934432 21510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:00:08.935521 21510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:00:08.935555 21510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:00:08.935567 21510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:00:08.935578 21510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:00:08.935588 21510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:00:08.937011 21510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:00:08.937209 21510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:00:08.937377 21510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:00:08.942019 21515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:08.942080 21516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:00:08.942107 21510 server_base.cc:1034] running on GCE node
W20260619 09:00:08.942019 21518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:00:08.942456 21510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:00:08.942726 21510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:00:08.943886 21510 hybrid_clock.cc:648] HybridClock initialized: now 1781859608943863 us; error 60 us; skew 500 ppm
Jun 19 09:00:08 dist-test-slave-h4g5 krb5kdc[21497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:00:08.946249 21510 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 09:00:08.946688 21510 webserver.cc:460] Webserver started at http://127.0.0.1:36405/ using document root <none> and password file <none>
I20260619 09:00:08.946892 21510 fs_manager.cc:359] Metadata directory not provided
I20260619 09:00:08.946940 21510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:00:08.947078 21510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:00:08.948184 21510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "96d2096688ea4a9d8826c762e75369b2"
format_stamp: "Formatted at 2026-06-19 09:00:08 on dist-test-slave-h4g5"
server_key: "ccfe75c717b3683187732f6ac0da8734"
server_key_iv: "e439360cca86f71d62e5fce478c0e018"
server_key_version: "encryptionkey@0"
I20260619 09:00:08.948511 21510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "96d2096688ea4a9d8826c762e75369b2"
format_stamp: "Formatted at 2026-06-19 09:00:08 on dist-test-slave-h4g5"
server_key: "ccfe75c717b3683187732f6ac0da8734"
server_key_iv: "e439360cca86f71d62e5fce478c0e018"
server_key_version: "encryptionkey@0"
I20260619 09:00:08.949688 21510 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:00:08.950348 21525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:00:08.950539 21510 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:00:08.950626 21510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "96d2096688ea4a9d8826c762e75369b2"
format_stamp: "Formatted at 2026-06-19 09:00:08 on dist-test-slave-h4g5"
server_key: "ccfe75c717b3683187732f6ac0da8734"
server_key_iv: "e439360cca86f71d62e5fce478c0e018"
server_key_version: "encryptionkey@0"
I20260619 09:00:08.950692 21510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:00:08.953809 21510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:00:08.955729 21510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:00:08.955828 21510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:00:08.959465 21510 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39585
I20260619 09:00:08.959528 21577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39585 every 8 connection(s)
I20260619 09:00:08.959805 21510 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:00:08.960783 21578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:00:08.960844 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 21510
I20260619 09:00:08.960970 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 09:00:08.961140 392 external_mini_cluster.cc:1428] Setting key e6d45fed3d99421bad590540eaf0ad1e
I20260619 09:00:08.963397 21581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:08.961609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:55249 (local address 127.0.0.1:39585)
0619 09:00:08.961775 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:08.961782 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:08.961807 (+ 25us) server_negotiation.cc:407] Connection header received
0619 09:00:08.962237 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:08.962245 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:08.962271 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:08.962380 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:08.962992 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:08.962995 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:08.962995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:00:08.962999 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:08.963034 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:00:08.963044 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:08.963097 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:08.963177 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 09:00:08.963213 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260619 09:00:08.963937 21578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96d2096688ea4a9d8826c762e75369b2: Bootstrap starting.
I20260619 09:00:08.964514 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:00:08.964578 21578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96d2096688ea4a9d8826c762e75369b2: Neither blocks nor log segments found. Creating new log.
I20260619 09:00:08.964839 21578 log.cc:824] T 00000000000000000000000000000000 P 96d2096688ea4a9d8826c762e75369b2: Log is configured to *not* fsync() on all Append() calls
I20260619 09:00:08.965435 21578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96d2096688ea4a9d8826c762e75369b2: No bootstrap required, opened a new log
I20260619 09:00:08.966888 21581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:08.965938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:36471 (local address 127.0.0.1:39585)
0619 09:00:08.965975 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:08.965977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:08.966015 (+ 38us) server_negotiation.cc:407] Connection header received
0619 09:00:08.966083 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:08.966083 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:08.966088 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:08.966130 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:08.966686 (+ 556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:08.966688 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:08.966688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:00:08.966691 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:08.966706 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:00:08.966710 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:08.966762 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:08.966822 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 09:00:08.966832 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 09:00:08.967604 21578 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96d2096688ea4a9d8826c762e75369b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96d2096688ea4a9d8826c762e75369b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39585 } }
I20260619 09:00:08.967738 21578 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96d2096688ea4a9d8826c762e75369b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:00:08.967770 21578 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96d2096688ea4a9d8826c762e75369b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96d2096688ea4a9d8826c762e75369b2, State: Initialized, Role: FOLLOWER
I20260619 09:00:08.967881 21578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96d2096688ea4a9d8826c762e75369b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96d2096688ea4a9d8826c762e75369b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39585 } }
I20260619 09:00:08.967952 21578 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96d2096688ea4a9d8826c762e75369b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:00:08.967983 21578 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96d2096688ea4a9d8826c762e75369b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:00:08.968029 21578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96d2096688ea4a9d8826c762e75369b2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:00:08.968559 21578 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96d2096688ea4a9d8826c762e75369b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96d2096688ea4a9d8826c762e75369b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39585 } }
I20260619 09:00:08.968650 21578 leader_election.cc:302] T 00000000000000000000000000000000 P 96d2096688ea4a9d8826c762e75369b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96d2096688ea4a9d8826c762e75369b2; no voters:
I20260619 09:00:08.968801 21578 leader_election.cc:288] T 00000000000000000000000000000000 P 96d2096688ea4a9d8826c762e75369b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:00:08.968850 21588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96d2096688ea4a9d8826c762e75369b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:00:08.968988 21588 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96d2096688ea4a9d8826c762e75369b2 [term 1 LEADER]: Becoming Leader. State: Replica: 96d2096688ea4a9d8826c762e75369b2, State: Running, Role: LEADER
I20260619 09:00:08.969025 21578 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96d2096688ea4a9d8826c762e75369b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:00:08.969108 21588 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96d2096688ea4a9d8826c762e75369b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96d2096688ea4a9d8826c762e75369b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39585 } }
I20260619 09:00:08.969468 21589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96d2096688ea4a9d8826c762e75369b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96d2096688ea4a9d8826c762e75369b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96d2096688ea4a9d8826c762e75369b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39585 } } }
I20260619 09:00:08.969477 21590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96d2096688ea4a9d8826c762e75369b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96d2096688ea4a9d8826c762e75369b2. Latest consensus state: current_term: 1 leader_uuid: "96d2096688ea4a9d8826c762e75369b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96d2096688ea4a9d8826c762e75369b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39585 } } }
I20260619 09:00:08.969619 21589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96d2096688ea4a9d8826c762e75369b2 [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:08.969638 21590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96d2096688ea4a9d8826c762e75369b2 [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:08.969833 21597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:00:08.970324 21597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:00:08.971792 21597 catalog_manager.cc:1347] Generated new cluster ID: 598a616910c6458b9f5980ed111530da
I20260619 09:00:08.971835 21597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:00:08.980777 21597 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:00:08.980932 21597 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:00:08.994059 21597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96d2096688ea4a9d8826c762e75369b2: Generated new TSK 0
I20260619 09:00:08.994271 21597 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:00:08.999091 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 21510
2026-06-19T09:00:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1202 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:00:09 dist-test-slave-h4g5 krb5kdc[21609](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:00:09 dist-test-slave-h4g5 krb5kdc[21609](info): set up 2 sockets
Jun 19 09:00:09 dist-test-slave-h4g5 krb5kdc[21609](info): commencing operation
krb5kdc: starting...
W20260619 09:00:09.951898 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 09:00:10 dist-test-slave-h4g5 krb5kdc[21609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:00:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:00:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 09:00:10.063468 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39685
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:00:10.145277 21622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:00:10.145503 21622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:00:10.145534 21622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:00:10.146690 21622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:00:10.146734 21622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:00:10.146754 21622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:00:10.146770 21622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:00:10.146785 21622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:00:10.148281 21622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39685
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:00:10.148535 21622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:00:10.148738 21622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:00:10.153532 21628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:10.153569 21630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:10.153527 21627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:00:10.153808 21622 server_base.cc:1034] running on GCE node
I20260619 09:00:10.154079 21622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:00:10.154354 21622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:00:10.155503 21622 hybrid_clock.cc:648] HybridClock initialized: now 1781859610155482 us; error 35 us; skew 500 ppm
Jun 19 09:00:10 dist-test-slave-h4g5 krb5kdc[21609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:00:10.158008 21622 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 09:00:10.158473 21622 webserver.cc:460] Webserver started at http://127.0.0.1:38469/ using document root <none> and password file <none>
I20260619 09:00:10.158672 21622 fs_manager.cc:359] Metadata directory not provided
I20260619 09:00:10.158722 21622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:00:10.158854 21622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:00:10.159891 21622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "aaba2260b86546fa88eb786a6df34e31"
format_stamp: "Formatted at 2026-06-19 09:00:10 on dist-test-slave-h4g5"
server_key: "df4fc3978ea62073c3dd379d9a461029"
server_key_iv: "700a40e913035843055bba70083d73ef"
server_key_version: "encryptionkey@0"
I20260619 09:00:10.160266 21622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "aaba2260b86546fa88eb786a6df34e31"
format_stamp: "Formatted at 2026-06-19 09:00:10 on dist-test-slave-h4g5"
server_key: "df4fc3978ea62073c3dd379d9a461029"
server_key_iv: "700a40e913035843055bba70083d73ef"
server_key_version: "encryptionkey@0"
I20260619 09:00:10.161507 21622 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 09:00:10.162266 21637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:00:10.162441 21622 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 09:00:10.162539 21622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "aaba2260b86546fa88eb786a6df34e31"
format_stamp: "Formatted at 2026-06-19 09:00:10 on dist-test-slave-h4g5"
server_key: "df4fc3978ea62073c3dd379d9a461029"
server_key_iv: "700a40e913035843055bba70083d73ef"
server_key_version: "encryptionkey@0"
I20260619 09:00:10.162600 21622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:00:10.181849 21622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:00:10.183882 21622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:00:10.184019 21622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:00:10.187742 21622 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39685
I20260619 09:00:10.187768 21689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39685 every 8 connection(s)
I20260619 09:00:10.188129 21622 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:00:10.188993 21690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:00:10.191233 21690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaba2260b86546fa88eb786a6df34e31: Bootstrap starting.
I20260619 09:00:10.191773 21690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aaba2260b86546fa88eb786a6df34e31: Neither blocks nor log segments found. Creating new log.
I20260619 09:00:10.192016 21690 log.cc:824] T 00000000000000000000000000000000 P aaba2260b86546fa88eb786a6df34e31: Log is configured to *not* fsync() on all Append() calls
I20260619 09:00:10.192698 21690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaba2260b86546fa88eb786a6df34e31: No bootstrap required, opened a new log
I20260619 09:00:10.194310 21690 raft_consensus.cc:348] T 00000000000000000000000000000000 P aaba2260b86546fa88eb786a6df34e31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaba2260b86546fa88eb786a6df34e31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39685 } }
I20260619 09:00:10.194451 21690 raft_consensus.cc:374] T 00000000000000000000000000000000 P aaba2260b86546fa88eb786a6df34e31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:00:10.194476 21690 raft_consensus.cc:729] T 00000000000000000000000000000000 P aaba2260b86546fa88eb786a6df34e31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaba2260b86546fa88eb786a6df34e31, State: Initialized, Role: FOLLOWER
I20260619 09:00:10.194588 21690 consensus_queue.cc:260] T 00000000000000000000000000000000 P aaba2260b86546fa88eb786a6df34e31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaba2260b86546fa88eb786a6df34e31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39685 } }
I20260619 09:00:10.194641 21690 raft_consensus.cc:388] T 00000000000000000000000000000000 P aaba2260b86546fa88eb786a6df34e31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:00:10.194666 21690 raft_consensus.cc:482] T 00000000000000000000000000000000 P aaba2260b86546fa88eb786a6df34e31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:00:10.194700 21690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aaba2260b86546fa88eb786a6df34e31 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:00:10.195191 21690 raft_consensus.cc:504] T 00000000000000000000000000000000 P aaba2260b86546fa88eb786a6df34e31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaba2260b86546fa88eb786a6df34e31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39685 } }
I20260619 09:00:10.195276 21690 leader_election.cc:302] T 00000000000000000000000000000000 P aaba2260b86546fa88eb786a6df34e31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aaba2260b86546fa88eb786a6df34e31; no voters:
I20260619 09:00:10.195403 21690 leader_election.cc:288] T 00000000000000000000000000000000 P aaba2260b86546fa88eb786a6df34e31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:00:10.195454 21693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aaba2260b86546fa88eb786a6df34e31 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:00:10.195629 21690 sys_catalog.cc:564] T 00000000000000000000000000000000 P aaba2260b86546fa88eb786a6df34e31 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:00:10.195617 21693 raft_consensus.cc:686] T 00000000000000000000000000000000 P aaba2260b86546fa88eb786a6df34e31 [term 1 LEADER]: Becoming Leader. State: Replica: aaba2260b86546fa88eb786a6df34e31, State: Running, Role: LEADER
I20260619 09:00:10.195757 21693 consensus_queue.cc:237] T 00000000000000000000000000000000 P aaba2260b86546fa88eb786a6df34e31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaba2260b86546fa88eb786a6df34e31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39685 } }
I20260619 09:00:10.196125 21695 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaba2260b86546fa88eb786a6df34e31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aaba2260b86546fa88eb786a6df34e31. Latest consensus state: current_term: 1 leader_uuid: "aaba2260b86546fa88eb786a6df34e31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaba2260b86546fa88eb786a6df34e31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39685 } } }
I20260619 09:00:10.196216 21695 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaba2260b86546fa88eb786a6df34e31 [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:10.196126 21694 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaba2260b86546fa88eb786a6df34e31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aaba2260b86546fa88eb786a6df34e31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaba2260b86546fa88eb786a6df34e31" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39685 } } }
I20260619 09:00:10.196256 21694 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaba2260b86546fa88eb786a6df34e31 [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:10.196415 21697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:00:10.196949 21697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:00:10.197975 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 21622
I20260619 09:00:10.198093 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 09:00:10.198297 392 external_mini_cluster.cc:1428] Setting key f565e9bda48c0a59e9f71db7b06c3a03
I20260619 09:00:10.198575 21697 catalog_manager.cc:1347] Generated new cluster ID: 00eb946cc6634bc080b15df312f8952a
I20260619 09:00:10.198628 21697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 09:00:10 dist-test-slave-h4g5 krb5kdc[21609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 09:00:10.208410 21697 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:00:10.209007 21697 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:00:10.210083 21713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:10.198937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:59431 (local address 127.0.0.1:39685)
0619 09:00:10.199084 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:10.199093 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:10.199123 (+ 30us) server_negotiation.cc:407] Connection header received
0619 09:00:10.199237 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:10.199244 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:10.199300 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:10.199450 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:10.200183 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:10.201631 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:10.202734 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:10.203335 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:10.203435 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:10.206326 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:10.206331 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:10.206331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:10.206343 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:10.208346 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:10.208817 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:10.208820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:10.208821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:10.208939 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:10.209156 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:10.209157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:10.209158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:10.209443 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:10.209576 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:10.209730 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 09:00:10.209836 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
I20260619 09:00:10.211369 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:00:10.214669 21697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aaba2260b86546fa88eb786a6df34e31: Generated new TSK 0
I20260619 09:00:10.214835 21697 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:00:10.219120 21713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:10.212804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:54975 (local address 127.0.0.1:39685)
0619 09:00:10.212837 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:10.212838 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:10.212982 (+ 144us) server_negotiation.cc:407] Connection header received
0619 09:00:10.213060 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:10.213061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:10.213070 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:10.213109 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:10.213781 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:10.214459 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:10.215308 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:10.215585 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:10.215642 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:10.216804 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:10.216805 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:10.216806 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:10.216812 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:10.218057 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:10.218391 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:10.218392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:10.218392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:10.218445 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:10.218658 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:10.218659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:10.218660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:10.218762 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:10.218847 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:10.218972 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 09:00:10.219035 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 09:00:10.221089 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 21622
W20260619 09:00:10.224642 21607 connection.cc:535] client connection to 127.0.0.1:39685 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39685): Connection reset by peer (error 104)
2026-06-19T09:00:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1221 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:00:10 dist-test-slave-h4g5 krb5kdc[21721](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:00:10 dist-test-slave-h4g5 krb5kdc[21721](info): set up 2 sockets
Jun 19 09:00:10 dist-test-slave-h4g5 krb5kdc[21721](info): commencing operation
krb5kdc: starting...
W20260619 09:00:11.179895 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 09:00:11 dist-test-slave-h4g5 krb5kdc[21721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:00:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:00:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 09:00:11.292402 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45053
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:00:11.377072 21734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:00:11.377277 21734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:00:11.377300 21734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:00:11.378461 21734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:00:11.378504 21734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:00:11.378515 21734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:00:11.378526 21734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:00:11.378537 21734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:00:11.379992 21734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45053
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:00:11.380257 21734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:00:11.380465 21734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:00:11.384974 21739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:11.385061 21740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:11.385039 21742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:00:11.385227 21734 server_base.cc:1034] running on GCE node
I20260619 09:00:11.385442 21734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:00:11.385692 21734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:00:11.386840 21734 hybrid_clock.cc:648] HybridClock initialized: now 1781859611386817 us; error 33 us; skew 500 ppm
Jun 19 09:00:11 dist-test-slave-h4g5 krb5kdc[21721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:00:11.389199 21734 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 09:00:11.389640 21734 webserver.cc:460] Webserver started at http://127.0.0.1:40807/ using document root <none> and password file <none>
I20260619 09:00:11.389850 21734 fs_manager.cc:359] Metadata directory not provided
I20260619 09:00:11.389897 21734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:00:11.390019 21734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:00:11.391016 21734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "428f43ffff414365a229851ca6bba537"
format_stamp: "Formatted at 2026-06-19 09:00:11 on dist-test-slave-h4g5"
server_key: "bb6093958fe4d27bf512c24a8613fa20"
server_key_iv: "51eee855b2b45954688860f78c54ada5"
server_key_version: "encryptionkey@0"
I20260619 09:00:11.391332 21734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "428f43ffff414365a229851ca6bba537"
format_stamp: "Formatted at 2026-06-19 09:00:11 on dist-test-slave-h4g5"
server_key: "bb6093958fe4d27bf512c24a8613fa20"
server_key_iv: "51eee855b2b45954688860f78c54ada5"
server_key_version: "encryptionkey@0"
I20260619 09:00:11.392565 21734 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 09:00:11.393177 21749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:00:11.393319 21734 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 09:00:11.393390 21734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "428f43ffff414365a229851ca6bba537"
format_stamp: "Formatted at 2026-06-19 09:00:11 on dist-test-slave-h4g5"
server_key: "bb6093958fe4d27bf512c24a8613fa20"
server_key_iv: "51eee855b2b45954688860f78c54ada5"
server_key_version: "encryptionkey@0"
I20260619 09:00:11.393453 21734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:00:11.404624 21734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:00:11.406461 21734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:00:11.406587 21734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:00:11.410313 21734 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45053
I20260619 09:00:11.410349 21801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45053 every 8 connection(s)
I20260619 09:00:11.410668 21734 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:00:11.411782 21802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:00:11.414062 21802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 428f43ffff414365a229851ca6bba537: Bootstrap starting.
I20260619 09:00:11.414577 21802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 428f43ffff414365a229851ca6bba537: Neither blocks nor log segments found. Creating new log.
I20260619 09:00:11.414793 21802 log.cc:824] T 00000000000000000000000000000000 P 428f43ffff414365a229851ca6bba537: Log is configured to *not* fsync() on all Append() calls
I20260619 09:00:11.415457 21802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 428f43ffff414365a229851ca6bba537: No bootstrap required, opened a new log
I20260619 09:00:11.416824 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 21734
I20260619 09:00:11.416960 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 09:00:11.417150 392 external_mini_cluster.cc:1428] Setting key 914ab9bfa5cef851df38e860ac39d00a
I20260619 09:00:11.417339 21802 raft_consensus.cc:348] T 00000000000000000000000000000000 P 428f43ffff414365a229851ca6bba537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "428f43ffff414365a229851ca6bba537" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45053 } }
I20260619 09:00:11.417495 21802 raft_consensus.cc:374] T 00000000000000000000000000000000 P 428f43ffff414365a229851ca6bba537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:00:11.417528 21802 raft_consensus.cc:729] T 00000000000000000000000000000000 P 428f43ffff414365a229851ca6bba537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428f43ffff414365a229851ca6bba537, State: Initialized, Role: FOLLOWER
I20260619 09:00:11.417682 21802 consensus_queue.cc:260] T 00000000000000000000000000000000 P 428f43ffff414365a229851ca6bba537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "428f43ffff414365a229851ca6bba537" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45053 } }
I20260619 09:00:11.417793 21802 raft_consensus.cc:388] T 00000000000000000000000000000000 P 428f43ffff414365a229851ca6bba537 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:00:11.417836 21802 raft_consensus.cc:482] T 00000000000000000000000000000000 P 428f43ffff414365a229851ca6bba537 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:00:11.417879 21802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 428f43ffff414365a229851ca6bba537 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:00:11.418720 21802 raft_consensus.cc:504] T 00000000000000000000000000000000 P 428f43ffff414365a229851ca6bba537 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "428f43ffff414365a229851ca6bba537" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45053 } }
I20260619 09:00:11.418814 21802 leader_election.cc:302] T 00000000000000000000000000000000 P 428f43ffff414365a229851ca6bba537 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 428f43ffff414365a229851ca6bba537; no voters:
I20260619 09:00:11.418967 21802 leader_election.cc:288] T 00000000000000000000000000000000 P 428f43ffff414365a229851ca6bba537 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:00:11.418999 21807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 428f43ffff414365a229851ca6bba537 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:00:11.419167 21807 raft_consensus.cc:686] T 00000000000000000000000000000000 P 428f43ffff414365a229851ca6bba537 [term 1 LEADER]: Becoming Leader. State: Replica: 428f43ffff414365a229851ca6bba537, State: Running, Role: LEADER
I20260619 09:00:11.419195 21802 sys_catalog.cc:564] T 00000000000000000000000000000000 P 428f43ffff414365a229851ca6bba537 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:00:11.419299 21807 consensus_queue.cc:237] T 00000000000000000000000000000000 P 428f43ffff414365a229851ca6bba537 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "428f43ffff414365a229851ca6bba537" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45053 } }
I20260619 09:00:11.419653 21808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 428f43ffff414365a229851ca6bba537 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "428f43ffff414365a229851ca6bba537" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "428f43ffff414365a229851ca6bba537" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45053 } } }
I20260619 09:00:11.419726 21808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 428f43ffff414365a229851ca6bba537 [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:11.419926 21809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 428f43ffff414365a229851ca6bba537 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 428f43ffff414365a229851ca6bba537. Latest consensus state: current_term: 1 leader_uuid: "428f43ffff414365a229851ca6bba537" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "428f43ffff414365a229851ca6bba537" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45053 } } }
I20260619 09:00:11.420025 21809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 428f43ffff414365a229851ca6bba537 [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:11.420434 21814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:00:11.421020 21814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:00:11.423341 21806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:11.417619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:40235 (local address 127.0.0.1:45053)
0619 09:00:11.417824 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:11.417833 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:11.417870 (+ 37us) server_negotiation.cc:407] Connection header received
0619 09:00:11.417991 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:11.417999 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:11.418056 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:11.418180 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:11.418390 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:11.419571 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:11.420637 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:11.421172 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:11.421277 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:11.422476 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:11.422480 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:11.422481 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:00:11.422681 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:11.422723 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:00:11.422739 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:11.422824 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:11.423152 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 09:00:11.423182 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
I20260619 09:00:11.424158 21814 catalog_manager.cc:1347] Generated new cluster ID: 26639c35dc634197a5312a80361982c0
I20260619 09:00:11.424228 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:00:11.424252 21814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:00:11.429081 21806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:11.425584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:58685 (local address 127.0.0.1:45053)
0619 09:00:11.425615 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:11.425617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:11.425813 (+ 196us) server_negotiation.cc:407] Connection header received
0619 09:00:11.425846 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:11.425847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:11.425858 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:11.425924 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:11.426040 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:11.426755 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:11.427727 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:11.428047 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:11.428163 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:11.428836 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:11.428838 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:11.428838 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:00:11.428843 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:11.428859 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:00:11.428863 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:11.428925 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:11.428985 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 09:00:11.428998 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 09:00:11.435040 21814 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:00:11.435619 21814 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:00:11.441181 21814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 428f43ffff414365a229851ca6bba537: Generated new TSK 0
I20260619 09:00:11.441395 21814 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:00:11.443878 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 21734
W20260619 09:00:11.447582 21719 connection.cc:535] client connection to 127.0.0.1:45053 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45053): Connection reset by peer (error 104)
2026-06-19T09:00:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1223 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:00:11 dist-test-slave-h4g5 krb5kdc[21833](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:00:11 dist-test-slave-h4g5 krb5kdc[21833](info): set up 2 sockets
Jun 19 09:00:11 dist-test-slave-h4g5 krb5kdc[21833](info): commencing operation
krb5kdc: starting...
W20260619 09:00:12.395895 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 09:00:12 dist-test-slave-h4g5 krb5kdc[21833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:00:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:00:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 09:00:12.507059 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34559
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:00:12.586455 21846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:00:12.586671 21846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:00:12.586702 21846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:00:12.587844 21846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:00:12.587886 21846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:00:12.587904 21846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:00:12.587920 21846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:00:12.587935 21846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:00:12.589488 21846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34559
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859361540516-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:00:12.589710 21846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:00:12.589897 21846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:00:12.594480 21854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:12.594485 21851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:12.594537 21852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:00:12.594528 21846 server_base.cc:1034] running on GCE node
I20260619 09:00:12.594895 21846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:00:12.595136 21846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:00:12.596285 21846 hybrid_clock.cc:648] HybridClock initialized: now 1781859612596268 us; error 37 us; skew 500 ppm
Jun 19 09:00:12 dist-test-slave-h4g5 krb5kdc[21833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:00:12.604027 21846 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 09:00:12.604519 21846 webserver.cc:460] Webserver started at http://127.0.0.1:40545/ using document root <none> and password file <none>
I20260619 09:00:12.604743 21846 fs_manager.cc:359] Metadata directory not provided
I20260619 09:00:12.604792 21846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:00:12.604915 21846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:00:12.605931 21846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "ef10e70cd8344794ad04d785cc3dbdc1"
format_stamp: "Formatted at 2026-06-19 09:00:12 on dist-test-slave-h4g5"
server_key: "50555e268fd1dd32d33c7f7aea05937c"
server_key_iv: "821936a424d64732a3496b7857742625"
server_key_version: "encryptionkey@0"
I20260619 09:00:12.606245 21846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "ef10e70cd8344794ad04d785cc3dbdc1"
format_stamp: "Formatted at 2026-06-19 09:00:12 on dist-test-slave-h4g5"
server_key: "50555e268fd1dd32d33c7f7aea05937c"
server_key_iv: "821936a424d64732a3496b7857742625"
server_key_version: "encryptionkey@0"
I20260619 09:00:12.607450 21846 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 09:00:12.608062 21861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:00:12.608248 21846 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 09:00:12.608337 21846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "ef10e70cd8344794ad04d785cc3dbdc1"
format_stamp: "Formatted at 2026-06-19 09:00:12 on dist-test-slave-h4g5"
server_key: "50555e268fd1dd32d33c7f7aea05937c"
server_key_iv: "821936a424d64732a3496b7857742625"
server_key_version: "encryptionkey@0"
I20260619 09:00:12.608402 21846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:00:12.611585 21846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:00:12.613487 21846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:00:12.613618 21846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:00:12.617352 21846 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34559
I20260619 09:00:12.617381 21913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34559 every 8 connection(s)
I20260619 09:00:12.617674 21846 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:00:12.618556 21914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:00:12.620761 21914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef10e70cd8344794ad04d785cc3dbdc1: Bootstrap starting.
I20260619 09:00:12.621321 21914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef10e70cd8344794ad04d785cc3dbdc1: Neither blocks nor log segments found. Creating new log.
I20260619 09:00:12.621385 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 21846
I20260619 09:00:12.621481 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 09:00:12.621599 21914 log.cc:824] T 00000000000000000000000000000000 P ef10e70cd8344794ad04d785cc3dbdc1: Log is configured to *not* fsync() on all Append() calls
I20260619 09:00:12.621642 392 external_mini_cluster.cc:1428] Setting key 7a7f740ca5fbf718f9165550c02fb956
I20260619 09:00:12.622292 21914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef10e70cd8344794ad04d785cc3dbdc1: No bootstrap required, opened a new log
I20260619 09:00:12.624258 21918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:12.622093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:54867 (local address 127.0.0.1:34559)
0619 09:00:12.622286 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:12.622295 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:12.622339 (+ 44us) server_negotiation.cc:407] Connection header received
0619 09:00:12.622787 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:12.622794 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:12.622820 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:12.622945 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:12.623675 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:12.623679 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:12.623679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:00:12.623683 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:12.623722 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:00:12.623732 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:12.623871 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:12.623905 (+ 34us) server_negotiation.cc:299] Negotiation successful
0619 09:00:12.623948 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":85,"threads_started":1}
I20260619 09:00:12.624274 21914 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef10e70cd8344794ad04d785cc3dbdc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef10e70cd8344794ad04d785cc3dbdc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34559 } }
I20260619 09:00:12.624464 21914 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef10e70cd8344794ad04d785cc3dbdc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:00:12.624501 21914 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef10e70cd8344794ad04d785cc3dbdc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef10e70cd8344794ad04d785cc3dbdc1, State: Initialized, Role: FOLLOWER
I20260619 09:00:12.624625 21914 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef10e70cd8344794ad04d785cc3dbdc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef10e70cd8344794ad04d785cc3dbdc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34559 } }
I20260619 09:00:12.624712 21914 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef10e70cd8344794ad04d785cc3dbdc1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:00:12.624804 21914 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef10e70cd8344794ad04d785cc3dbdc1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:00:12.624845 21914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef10e70cd8344794ad04d785cc3dbdc1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:00:12.625144 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:00:12.625494 21914 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef10e70cd8344794ad04d785cc3dbdc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef10e70cd8344794ad04d785cc3dbdc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34559 } }
I20260619 09:00:12.625587 21914 leader_election.cc:302] T 00000000000000000000000000000000 P ef10e70cd8344794ad04d785cc3dbdc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef10e70cd8344794ad04d785cc3dbdc1; no voters:
I20260619 09:00:12.625754 21914 leader_election.cc:288] T 00000000000000000000000000000000 P ef10e70cd8344794ad04d785cc3dbdc1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:00:12.625795 21919 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef10e70cd8344794ad04d785cc3dbdc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:00:12.625927 21919 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef10e70cd8344794ad04d785cc3dbdc1 [term 1 LEADER]: Becoming Leader. State: Replica: ef10e70cd8344794ad04d785cc3dbdc1, State: Running, Role: LEADER
I20260619 09:00:12.626016 21914 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef10e70cd8344794ad04d785cc3dbdc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:00:12.626101 21919 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef10e70cd8344794ad04d785cc3dbdc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef10e70cd8344794ad04d785cc3dbdc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34559 } }
I20260619 09:00:12.626466 21921 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef10e70cd8344794ad04d785cc3dbdc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef10e70cd8344794ad04d785cc3dbdc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef10e70cd8344794ad04d785cc3dbdc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34559 } } }
I20260619 09:00:12.626549 21921 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef10e70cd8344794ad04d785cc3dbdc1 [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:12.626673 21921 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef10e70cd8344794ad04d785cc3dbdc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef10e70cd8344794ad04d785cc3dbdc1. Latest consensus state: current_term: 1 leader_uuid: "ef10e70cd8344794ad04d785cc3dbdc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef10e70cd8344794ad04d785cc3dbdc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34559 } } }
I20260619 09:00:12.626716 21921 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef10e70cd8344794ad04d785cc3dbdc1 [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:12.627007 21932 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:00:12.627477 21932 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:00:12.629056 21918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:12.627854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.137:41685 (local address 127.0.0.1:34559)
0619 09:00:12.627886 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:12.627888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:12.628021 (+ 133us) server_negotiation.cc:407] Connection header received
0619 09:00:12.628153 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:12.628155 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:12.628160 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:12.628222 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:12.628841 (+ 619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:12.628842 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:12.628842 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:00:12.628846 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:12.628859 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:00:12.628863 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:12.628889 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:12.628987 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 09:00:12.629000 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 09:00:12.629092 21932 catalog_manager.cc:1347] Generated new cluster ID: 029a38467da647929ac063e745010124
I20260619 09:00:12.629133 21932 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:00:12.650867 21932 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:00:12.651074 21932 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:00:12.656183 21932 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef10e70cd8344794ad04d785cc3dbdc1: Generated new TSK 0
I20260619 09:00:12.656397 21932 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:00:12.665814 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 21846
2026-06-19T09:00:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1221 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:00:12 dist-test-slave-h4g5 krb5kdc[21945](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:00:12 dist-test-slave-h4g5 krb5kdc[21945](info): set up 2 sockets
Jun 19 09:00:12 dist-test-slave-h4g5 krb5kdc[21945](info): commencing operation
krb5kdc: starting...
W20260619 09:00:13.615975 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 09:00:13 dist-test-slave-h4g5 krb5kdc[21945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:00:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:00:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 09:00:13.726563 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46665
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:00:13.805302 21958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:00:13.805507 21958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:00:13.805529 21958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:00:13.806648 21958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:00:13.806684 21958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:00:13.806696 21958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:00:13.806707 21958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:00:13.806720 21958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:00:13.808586 21958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46665
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:00:13.808806 21958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:00:13.808980 21958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:00:13.813547 21964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:13.813542 21963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:13.813524 21966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:00:13.813843 21958 server_base.cc:1034] running on GCE node
I20260619 09:00:13.814035 21958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:00:13.814267 21958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:00:13.815412 21958 hybrid_clock.cc:648] HybridClock initialized: now 1781859613815396 us; error 28 us; skew 500 ppm
Jun 19 09:00:13 dist-test-slave-h4g5 krb5kdc[21945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:00:13.817739 21958 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:00:13.818138 21958 webserver.cc:460] Webserver started at http://127.0.98.62:38319/ using document root <none> and password file <none>
I20260619 09:00:13.818353 21958 fs_manager.cc:359] Metadata directory not provided
I20260619 09:00:13.818401 21958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:00:13.818519 21958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:00:13.819556 21958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "5f736c6839284f949653cfc134722caf"
format_stamp: "Formatted at 2026-06-19 09:00:13 on dist-test-slave-h4g5"
server_key: "49501eb61624194ad7f4d3251338cdb4"
server_key_iv: "45693374bef521a1dcd2a654dc855ad2"
server_key_version: "encryptionkey@0"
I20260619 09:00:13.819882 21958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "5f736c6839284f949653cfc134722caf"
format_stamp: "Formatted at 2026-06-19 09:00:13 on dist-test-slave-h4g5"
server_key: "49501eb61624194ad7f4d3251338cdb4"
server_key_iv: "45693374bef521a1dcd2a654dc855ad2"
server_key_version: "encryptionkey@0"
I20260619 09:00:13.821147 21958 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:00:13.821796 21973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:00:13.821950 21958 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:00:13.822027 21958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "5f736c6839284f949653cfc134722caf"
format_stamp: "Formatted at 2026-06-19 09:00:13 on dist-test-slave-h4g5"
server_key: "49501eb61624194ad7f4d3251338cdb4"
server_key_iv: "45693374bef521a1dcd2a654dc855ad2"
server_key_version: "encryptionkey@0"
I20260619 09:00:13.822093 21958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:00:13.838101 21958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:00:13.840022 21958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:00:13.840191 21958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:00:13.843772 21958 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46665
I20260619 09:00:13.843817 22025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46665 every 8 connection(s)
I20260619 09:00:13.844138 21958 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:00:13.844959 22026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:00:13.847158 22026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f736c6839284f949653cfc134722caf: Bootstrap starting.
I20260619 09:00:13.847689 22026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f736c6839284f949653cfc134722caf: Neither blocks nor log segments found. Creating new log.
I20260619 09:00:13.847942 22026 log.cc:824] T 00000000000000000000000000000000 P 5f736c6839284f949653cfc134722caf: Log is configured to *not* fsync() on all Append() calls
I20260619 09:00:13.848608 22026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f736c6839284f949653cfc134722caf: No bootstrap required, opened a new log
I20260619 09:00:13.850142 22026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f736c6839284f949653cfc134722caf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f736c6839284f949653cfc134722caf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } }
I20260619 09:00:13.850281 22026 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f736c6839284f949653cfc134722caf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:00:13.850317 22026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f736c6839284f949653cfc134722caf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f736c6839284f949653cfc134722caf, State: Initialized, Role: FOLLOWER
I20260619 09:00:13.850456 22026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f736c6839284f949653cfc134722caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f736c6839284f949653cfc134722caf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } }
I20260619 09:00:13.850518 22026 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f736c6839284f949653cfc134722caf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:00:13.850539 22026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f736c6839284f949653cfc134722caf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:00:13.850612 22026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f736c6839284f949653cfc134722caf [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:00:13.851106 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 21958
I20260619 09:00:13.851230 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 09:00:13.851239 22026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f736c6839284f949653cfc134722caf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f736c6839284f949653cfc134722caf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } }
I20260619 09:00:13.851342 22026 leader_election.cc:302] T 00000000000000000000000000000000 P 5f736c6839284f949653cfc134722caf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f736c6839284f949653cfc134722caf; no voters:
I20260619 09:00:13.851410 392 external_mini_cluster.cc:1428] Setting key 637a349c3c0e3360fddef90f3912e79e
I20260619 09:00:13.851512 22026 leader_election.cc:288] T 00000000000000000000000000000000 P 5f736c6839284f949653cfc134722caf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:00:13.851538 22029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f736c6839284f949653cfc134722caf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:00:13.851707 22029 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f736c6839284f949653cfc134722caf [term 1 LEADER]: Becoming Leader. State: Replica: 5f736c6839284f949653cfc134722caf, State: Running, Role: LEADER
I20260619 09:00:13.851764 22026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f736c6839284f949653cfc134722caf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:00:13.851814 22029 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f736c6839284f949653cfc134722caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f736c6839284f949653cfc134722caf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } }
I20260619 09:00:13.852173 22029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f736c6839284f949653cfc134722caf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f736c6839284f949653cfc134722caf. Latest consensus state: current_term: 1 leader_uuid: "5f736c6839284f949653cfc134722caf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f736c6839284f949653cfc134722caf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } }
I20260619 09:00:13.852260 22029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f736c6839284f949653cfc134722caf [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:13.852504 22038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:00:13.852483 22030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f736c6839284f949653cfc134722caf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f736c6839284f949653cfc134722caf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f736c6839284f949653cfc134722caf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46665 } } }
I20260619 09:00:13.852618 22030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f736c6839284f949653cfc134722caf [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:13.853163 22038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:00:13.854964 22038 catalog_manager.cc:1347] Generated new cluster ID: b909db92fd044e67bd476f290482228c
I20260619 09:00:13.855015 22038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 09:00:13 dist-test-slave-h4g5 krb5kdc[21945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:00:13.861732 22034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:13.852112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54403 (local address 127.0.98.62:46665)
0619 09:00:13.852376 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:13.852388 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:13.852422 (+ 34us) server_negotiation.cc:407] Connection header received
0619 09:00:13.852542 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:13.852550 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:13.852608 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:13.852774 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:13.853549 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:13.854697 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:13.855528 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:13.855963 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:13.856072 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:13.858583 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:13.858588 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:13.858589 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:13.858603 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:13.860192 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:13.860563 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:13.860566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:13.860566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:13.860666 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:13.860867 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:13.860868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:13.860869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:13.861121 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:13.861275 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:13.861400 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 09:00:13.861500 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":61,"threads_started":1}
I20260619 09:00:13.862685 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:00:13.869572 22034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:13.863933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58873 (local address 127.0.98.62:46665)
0619 09:00:13.863969 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:13.863971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:13.864255 (+ 284us) server_negotiation.cc:407] Connection header received
0619 09:00:13.864307 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:13.864308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:13.864319 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:13.864369 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:13.864838 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:13.865317 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:13.865958 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:13.866186 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:13.866247 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:13.867328 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:13.867329 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:13.867329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:13.867336 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:13.868558 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:13.868878 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:13.868879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:13.868880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:13.868930 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:13.869110 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:13.869111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:13.869111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:13.869217 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:13.869302 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:13.869416 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 09:00:13.869483 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 09:00:13.877353 22038 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:00:13.877867 22038 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:00:13.885509 22038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f736c6839284f949653cfc134722caf: Generated new TSK 0
I20260619 09:00:13.885710 22038 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:00:13.892935 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 21958
W20260619 09:00:13.896800 21943 connection.cc:535] client connection to 127.0.98.62:46665 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:46665): Connection reset by peer (error 104)
2026-06-19T09:00:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1227 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:00:13 dist-test-slave-h4g5 krb5kdc[22057](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:00:13 dist-test-slave-h4g5 krb5kdc[22057](info): set up 2 sockets
Jun 19 09:00:13 dist-test-slave-h4g5 krb5kdc[22057](info): commencing operation
krb5kdc: starting...
W20260619 09:00:14.840020 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 09:00:14 dist-test-slave-h4g5 krb5kdc[22057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:00:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:00:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 09:00:14.950209 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34597
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:00:15.028589 22070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:00:15.028796 22070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:00:15.028818 22070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:00:15.029973 22070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:00:15.030016 22070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:00:15.030030 22070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:00:15.030040 22070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:00:15.030050 22070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:00:15.031584 22070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34597
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:00:15.031778 22070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:00:15.031939 22070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:00:15.036736 22078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:15.036811 22076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:00:15.036779 22070 server_base.cc:1034] running on GCE node
W20260619 09:00:15.036743 22075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:00:15.037184 22070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:00:15.037473 22070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:00:15.038638 22070 hybrid_clock.cc:648] HybridClock initialized: now 1781859615038633 us; error 72 us; skew 500 ppm
Jun 19 09:00:15 dist-test-slave-h4g5 krb5kdc[22057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:00:15.041244 22070 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:00:15.041671 22070 webserver.cc:460] Webserver started at http://127.0.98.62:44229/ using document root <none> and password file <none>
I20260619 09:00:15.041867 22070 fs_manager.cc:359] Metadata directory not provided
I20260619 09:00:15.041914 22070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:00:15.042035 22070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:00:15.043093 22070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "3a51856ee5bb41c18f38c1332bbe718e"
format_stamp: "Formatted at 2026-06-19 09:00:15 on dist-test-slave-h4g5"
server_key: "44534ed4491c2a1eca3ff0bab50b0194"
server_key_iv: "e888c444a97d61fe07f25e5c97bcde1d"
server_key_version: "encryptionkey@0"
I20260619 09:00:15.043414 22070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "3a51856ee5bb41c18f38c1332bbe718e"
format_stamp: "Formatted at 2026-06-19 09:00:15 on dist-test-slave-h4g5"
server_key: "44534ed4491c2a1eca3ff0bab50b0194"
server_key_iv: "e888c444a97d61fe07f25e5c97bcde1d"
server_key_version: "encryptionkey@0"
I20260619 09:00:15.044741 22070 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:00:15.045405 22085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:00:15.045552 22070 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:00:15.045630 22070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "3a51856ee5bb41c18f38c1332bbe718e"
format_stamp: "Formatted at 2026-06-19 09:00:15 on dist-test-slave-h4g5"
server_key: "44534ed4491c2a1eca3ff0bab50b0194"
server_key_iv: "e888c444a97d61fe07f25e5c97bcde1d"
server_key_version: "encryptionkey@0"
I20260619 09:00:15.045725 22070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:00:15.057192 22070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:00:15.059487 22070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:00:15.059628 22070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:00:15.063347 22070 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34597
I20260619 09:00:15.063387 22137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34597 every 8 connection(s)
I20260619 09:00:15.063687 22070 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:00:15.064546 22138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:00:15.064726 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 22070
I20260619 09:00:15.064834 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 09:00:15.065001 392 external_mini_cluster.cc:1428] Setting key 6e7964fe63360034e015da909f212bbe
I20260619 09:00:15.067170 22138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a51856ee5bb41c18f38c1332bbe718e: Bootstrap starting.
I20260619 09:00:15.067721 22138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a51856ee5bb41c18f38c1332bbe718e: Neither blocks nor log segments found. Creating new log.
I20260619 09:00:15.068004 22138 log.cc:824] T 00000000000000000000000000000000 P 3a51856ee5bb41c18f38c1332bbe718e: Log is configured to *not* fsync() on all Append() calls
I20260619 09:00:15.068720 22138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a51856ee5bb41c18f38c1332bbe718e: No bootstrap required, opened a new log
I20260619 09:00:15.070519 22141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:15.065616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54347 (local address 127.0.98.62:34597)
0619 09:00:15.065800 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:15.065808 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:15.065839 (+ 31us) server_negotiation.cc:407] Connection header received
0619 09:00:15.066291 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:15.066298 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:15.066353 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:15.066479 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:15.066742 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:15.067680 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:15.068672 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:15.069083 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:15.069169 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:15.069917 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:15.069920 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:15.069921 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:00:15.070066 (+ 145us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:15.070095 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:00:15.070105 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:15.070183 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:15.070302 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 09:00:15.070328 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":74,"threads_started":1}
I20260619 09:00:15.070717 22138 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a51856ee5bb41c18f38c1332bbe718e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a51856ee5bb41c18f38c1332bbe718e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34597 } }
I20260619 09:00:15.070878 22138 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a51856ee5bb41c18f38c1332bbe718e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:00:15.070902 22138 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a51856ee5bb41c18f38c1332bbe718e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a51856ee5bb41c18f38c1332bbe718e, State: Initialized, Role: FOLLOWER
I20260619 09:00:15.071002 22138 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a51856ee5bb41c18f38c1332bbe718e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a51856ee5bb41c18f38c1332bbe718e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34597 } }
I20260619 09:00:15.071069 22138 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a51856ee5bb41c18f38c1332bbe718e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:00:15.071097 22138 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a51856ee5bb41c18f38c1332bbe718e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:00:15.071143 22138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a51856ee5bb41c18f38c1332bbe718e [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:00:15.071446 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:00:15.071787 22138 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a51856ee5bb41c18f38c1332bbe718e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a51856ee5bb41c18f38c1332bbe718e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34597 } }
I20260619 09:00:15.071887 22138 leader_election.cc:302] T 00000000000000000000000000000000 P 3a51856ee5bb41c18f38c1332bbe718e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a51856ee5bb41c18f38c1332bbe718e; no voters:
I20260619 09:00:15.072045 22138 leader_election.cc:288] T 00000000000000000000000000000000 P 3a51856ee5bb41c18f38c1332bbe718e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:00:15.072103 22143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a51856ee5bb41c18f38c1332bbe718e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:00:15.072243 22143 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a51856ee5bb41c18f38c1332bbe718e [term 1 LEADER]: Becoming Leader. State: Replica: 3a51856ee5bb41c18f38c1332bbe718e, State: Running, Role: LEADER
I20260619 09:00:15.072304 22138 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a51856ee5bb41c18f38c1332bbe718e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:00:15.072373 22143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a51856ee5bb41c18f38c1332bbe718e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a51856ee5bb41c18f38c1332bbe718e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34597 } }
I20260619 09:00:15.072726 22143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a51856ee5bb41c18f38c1332bbe718e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a51856ee5bb41c18f38c1332bbe718e. Latest consensus state: current_term: 1 leader_uuid: "3a51856ee5bb41c18f38c1332bbe718e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a51856ee5bb41c18f38c1332bbe718e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34597 } } }
I20260619 09:00:15.072747 22145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a51856ee5bb41c18f38c1332bbe718e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a51856ee5bb41c18f38c1332bbe718e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a51856ee5bb41c18f38c1332bbe718e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34597 } } }
I20260619 09:00:15.072834 22143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a51856ee5bb41c18f38c1332bbe718e [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:15.072844 22145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a51856ee5bb41c18f38c1332bbe718e [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:15.073122 22156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:00:15.073604 22156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:00:15.075393 22156 catalog_manager.cc:1347] Generated new cluster ID: f80c821a754642e291a42a5a6bddc23c
I20260619 09:00:15.075435 22156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:00:15.077395 22141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:15.073812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39781 (local address 127.0.98.62:34597)
0619 09:00:15.073845 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:15.073846 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:15.074109 (+ 263us) server_negotiation.cc:407] Connection header received
0619 09:00:15.074155 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:15.074156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:15.074166 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:15.074219 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:15.074403 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:15.075006 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:15.076009 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:15.076350 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:15.076393 (+ 43us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:15.077120 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:15.077122 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:15.077122 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:00:15.077136 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:15.077149 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:00:15.077153 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:15.077185 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:15.077314 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 09:00:15.077325 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 09:00:15.096895 22156 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:00:15.097416 22156 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:00:15.105022 22156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a51856ee5bb41c18f38c1332bbe718e: Generated new TSK 0
I20260619 09:00:15.105245 22156 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:00:15.107983 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 22070
W20260619 09:00:15.111766 22055 connection.cc:535] client connection to 127.0.98.62:34597 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:34597): Connection reset by peer (error 104)
2026-06-19T09:00:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1214 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:00:15 dist-test-slave-h4g5 krb5kdc[22169](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:00:15 dist-test-slave-h4g5 krb5kdc[22169](info): set up 2 sockets
Jun 19 09:00:15 dist-test-slave-h4g5 krb5kdc[22169](info): commencing operation
krb5kdc: starting...
W20260619 09:00:16.052001 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 09:00:16 dist-test-slave-h4g5 krb5kdc[22169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:00:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:00:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 09:00:16.167227 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34679
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:00:16.246606 22182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:00:16.246830 22182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:00:16.246863 22182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:00:16.248056 22182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:00:16.248127 22182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:00:16.248147 22182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:00:16.248162 22182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:00:16.248175 22182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:00:16.249709 22182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34679
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:00:16.249955 22182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:00:16.250176 22182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:00:16.255137 22187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:16.255160 22190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:16.255165 22188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:00:16.255348 22182 server_base.cc:1034] running on GCE node
I20260619 09:00:16.255586 22182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:00:16.255878 22182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:00:16.257016 22182 hybrid_clock.cc:648] HybridClock initialized: now 1781859616256997 us; error 42 us; skew 500 ppm
Jun 19 09:00:16 dist-test-slave-h4g5 krb5kdc[22169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:00:16.259359 22182 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:00:16.259794 22182 webserver.cc:460] Webserver started at http://127.0.98.62:38287/ using document root <none> and password file <none>
I20260619 09:00:16.260004 22182 fs_manager.cc:359] Metadata directory not provided
I20260619 09:00:16.260072 22182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:00:16.260221 22182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:00:16.261273 22182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "0857bbbd72c743b6b0e5bf4d6118748f"
format_stamp: "Formatted at 2026-06-19 09:00:16 on dist-test-slave-h4g5"
server_key: "644de63c4c56fab154c3bd850ae3e5c9"
server_key_iv: "f76d809440fc27b53c7b676145dd6564"
server_key_version: "encryptionkey@0"
I20260619 09:00:16.261652 22182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "0857bbbd72c743b6b0e5bf4d6118748f"
format_stamp: "Formatted at 2026-06-19 09:00:16 on dist-test-slave-h4g5"
server_key: "644de63c4c56fab154c3bd850ae3e5c9"
server_key_iv: "f76d809440fc27b53c7b676145dd6564"
server_key_version: "encryptionkey@0"
I20260619 09:00:16.262952 22182 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 09:00:16.263666 22197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:00:16.263831 22182 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:00:16.263919 22182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "0857bbbd72c743b6b0e5bf4d6118748f"
format_stamp: "Formatted at 2026-06-19 09:00:16 on dist-test-slave-h4g5"
server_key: "644de63c4c56fab154c3bd850ae3e5c9"
server_key_iv: "f76d809440fc27b53c7b676145dd6564"
server_key_version: "encryptionkey@0"
I20260619 09:00:16.264001 22182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:00:16.267338 22182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:00:16.269490 22182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:00:16.269626 22182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:00:16.273573 22182 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34679
I20260619 09:00:16.273622 22249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34679 every 8 connection(s)
I20260619 09:00:16.273929 22182 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:00:16.274899 22250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:00:16.277185 22250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0857bbbd72c743b6b0e5bf4d6118748f: Bootstrap starting.
I20260619 09:00:16.277730 22250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0857bbbd72c743b6b0e5bf4d6118748f: Neither blocks nor log segments found. Creating new log.
I20260619 09:00:16.277976 22250 log.cc:824] T 00000000000000000000000000000000 P 0857bbbd72c743b6b0e5bf4d6118748f: Log is configured to *not* fsync() on all Append() calls
I20260619 09:00:16.278635 22250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0857bbbd72c743b6b0e5bf4d6118748f: No bootstrap required, opened a new log
I20260619 09:00:16.280228 22250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0857bbbd72c743b6b0e5bf4d6118748f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0857bbbd72c743b6b0e5bf4d6118748f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34679 } }
I20260619 09:00:16.280370 22250 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0857bbbd72c743b6b0e5bf4d6118748f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:00:16.280392 22250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0857bbbd72c743b6b0e5bf4d6118748f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0857bbbd72c743b6b0e5bf4d6118748f, State: Initialized, Role: FOLLOWER
I20260619 09:00:16.280519 22250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0857bbbd72c743b6b0e5bf4d6118748f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0857bbbd72c743b6b0e5bf4d6118748f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34679 } }
I20260619 09:00:16.280587 22250 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0857bbbd72c743b6b0e5bf4d6118748f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:00:16.280608 22250 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0857bbbd72c743b6b0e5bf4d6118748f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:00:16.280639 22250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0857bbbd72c743b6b0e5bf4d6118748f [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:00:16.281140 22250 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0857bbbd72c743b6b0e5bf4d6118748f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0857bbbd72c743b6b0e5bf4d6118748f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34679 } }
I20260619 09:00:16.281226 22250 leader_election.cc:302] T 00000000000000000000000000000000 P 0857bbbd72c743b6b0e5bf4d6118748f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0857bbbd72c743b6b0e5bf4d6118748f; no voters:
I20260619 09:00:16.281376 22250 leader_election.cc:288] T 00000000000000000000000000000000 P 0857bbbd72c743b6b0e5bf4d6118748f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:00:16.281414 22253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0857bbbd72c743b6b0e5bf4d6118748f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:00:16.281622 22250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0857bbbd72c743b6b0e5bf4d6118748f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:00:16.281766 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 22182
I20260619 09:00:16.281888 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 09:00:16.282048 392 external_mini_cluster.cc:1428] Setting key 4e67cc16667cd09b7ee997af20c9cfe3
I20260619 09:00:16.281569 22253 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0857bbbd72c743b6b0e5bf4d6118748f [term 1 LEADER]: Becoming Leader. State: Replica: 0857bbbd72c743b6b0e5bf4d6118748f, State: Running, Role: LEADER
I20260619 09:00:16.282254 22253 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0857bbbd72c743b6b0e5bf4d6118748f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0857bbbd72c743b6b0e5bf4d6118748f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34679 } }
I20260619 09:00:16.282847 22254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0857bbbd72c743b6b0e5bf4d6118748f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0857bbbd72c743b6b0e5bf4d6118748f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0857bbbd72c743b6b0e5bf4d6118748f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34679 } } }
I20260619 09:00:16.282946 22254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0857bbbd72c743b6b0e5bf4d6118748f [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:16.283032 22255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0857bbbd72c743b6b0e5bf4d6118748f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0857bbbd72c743b6b0e5bf4d6118748f. Latest consensus state: current_term: 1 leader_uuid: "0857bbbd72c743b6b0e5bf4d6118748f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0857bbbd72c743b6b0e5bf4d6118748f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34679 } } }
I20260619 09:00:16.283082 22255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0857bbbd72c743b6b0e5bf4d6118748f [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:16.283181 22270 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:00:16.283818 22270 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:00:16.284451 22267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:16.282723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59621 (local address 127.0.98.62:34679)
0619 09:00:16.282880 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:16.282890 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:16.282923 (+ 33us) server_negotiation.cc:407] Connection header received
0619 09:00:16.283052 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:16.283058 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:16.283086 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:16.283229 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:16.283979 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:16.283983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:16.283983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:00:16.283988 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:16.284028 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:00:16.284057 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:16.284167 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:16.284229 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 09:00:16.284271 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":56,"threads_started":1}
I20260619 09:00:16.285300 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:00:16.285461 22270 catalog_manager.cc:1347] Generated new cluster ID: 4e585fe0686c45329ca4c7e748ba957d
I20260619 09:00:16.285503 22270 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:00:16.293165 22270 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:00:16.293331 22270 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:00:16.293385 22267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:16.288368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34121 (local address 127.0.98.62:34679)
0619 09:00:16.288403 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:16.288405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:16.292289 (+ 3884us) server_negotiation.cc:407] Connection header received
0619 09:00:16.292362 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:16.292363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:16.292370 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:16.292418 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:16.293070 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:16.293072 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:16.293072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:00:16.293077 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:16.293092 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:00:16.293097 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:16.293143 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:16.293270 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 09:00:16.293283 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 09:00:16.296402 22270 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0857bbbd72c743b6b0e5bf4d6118748f: Generated new TSK 0
I20260619 09:00:16.296592 22270 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:00:16.298468 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 22182
2026-06-19T09:00:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1190 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:00:16 dist-test-slave-h4g5 krb5kdc[22281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:00:16 dist-test-slave-h4g5 krb5kdc[22281](info): set up 2 sockets
Jun 19 09:00:16 dist-test-slave-h4g5 krb5kdc[22281](info): commencing operation
krb5kdc: starting...
W20260619 09:00:17.247964 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 09:00:17 dist-test-slave-h4g5 krb5kdc[22281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:00:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:00:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 09:00:17.359475 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40745
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:00:17.437330 22294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:00:17.437526 22294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:00:17.437548 22294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:00:17.438669 22294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:00:17.438709 22294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:00:17.438721 22294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:00:17.438733 22294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:00:17.438755 22294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:00:17.440199 22294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:00:17.440414 22294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:00:17.440613 22294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:00:17.445689 22300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:00:17.445694 22294 server_base.cc:1034] running on GCE node
W20260619 09:00:17.445802 22302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:17.445919 22299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:00:17.446159 22294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:00:17.446478 22294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:00:17.447638 22294 hybrid_clock.cc:648] HybridClock initialized: now 1781859617447615 us; error 39 us; skew 500 ppm
Jun 19 09:00:17 dist-test-slave-h4g5 krb5kdc[22281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:00:17.449934 22294 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:00:17.450346 22294 webserver.cc:460] Webserver started at http://127.0.98.62:41223/ using document root <none> and password file <none>
I20260619 09:00:17.450575 22294 fs_manager.cc:359] Metadata directory not provided
I20260619 09:00:17.450623 22294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:00:17.450743 22294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:00:17.451754 22294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "06d313b6f411460996d911803d6a1a0e"
format_stamp: "Formatted at 2026-06-19 09:00:17 on dist-test-slave-h4g5"
server_key: "3a46b0a4eed202f81735a288af026aab"
server_key_iv: "61fbdf289c2c06fd4d86079aa6cbf5a9"
server_key_version: "encryptionkey@0"
I20260619 09:00:17.452069 22294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "06d313b6f411460996d911803d6a1a0e"
format_stamp: "Formatted at 2026-06-19 09:00:17 on dist-test-slave-h4g5"
server_key: "3a46b0a4eed202f81735a288af026aab"
server_key_iv: "61fbdf289c2c06fd4d86079aa6cbf5a9"
server_key_version: "encryptionkey@0"
I20260619 09:00:17.453316 22294 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 09:00:17.454064 22309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:00:17.454221 22294 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:00:17.454306 22294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "06d313b6f411460996d911803d6a1a0e"
format_stamp: "Formatted at 2026-06-19 09:00:17 on dist-test-slave-h4g5"
server_key: "3a46b0a4eed202f81735a288af026aab"
server_key_iv: "61fbdf289c2c06fd4d86079aa6cbf5a9"
server_key_version: "encryptionkey@0"
I20260619 09:00:17.454376 22294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:00:17.467703 22294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:00:17.469699 22294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:00:17.469835 22294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:00:17.473604 22294 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40745
I20260619 09:00:17.473639 22361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40745 every 8 connection(s)
I20260619 09:00:17.473951 22294 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:00:17.474758 22362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:00:17.476962 22362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06d313b6f411460996d911803d6a1a0e: Bootstrap starting.
I20260619 09:00:17.477487 22362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06d313b6f411460996d911803d6a1a0e: Neither blocks nor log segments found. Creating new log.
I20260619 09:00:17.477726 22362 log.cc:824] T 00000000000000000000000000000000 P 06d313b6f411460996d911803d6a1a0e: Log is configured to *not* fsync() on all Append() calls
I20260619 09:00:17.478346 22362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06d313b6f411460996d911803d6a1a0e: No bootstrap required, opened a new log
I20260619 09:00:17.479889 22362 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06d313b6f411460996d911803d6a1a0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d313b6f411460996d911803d6a1a0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40745 } }
I20260619 09:00:17.480026 22362 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06d313b6f411460996d911803d6a1a0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:00:17.480046 22362 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06d313b6f411460996d911803d6a1a0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06d313b6f411460996d911803d6a1a0e, State: Initialized, Role: FOLLOWER
I20260619 09:00:17.480202 22362 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06d313b6f411460996d911803d6a1a0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d313b6f411460996d911803d6a1a0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40745 } }
I20260619 09:00:17.480264 22362 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06d313b6f411460996d911803d6a1a0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:00:17.480284 22362 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06d313b6f411460996d911803d6a1a0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:00:17.480329 22362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06d313b6f411460996d911803d6a1a0e [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:00:17.480820 22362 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06d313b6f411460996d911803d6a1a0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d313b6f411460996d911803d6a1a0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40745 } }
I20260619 09:00:17.480902 22362 leader_election.cc:302] T 00000000000000000000000000000000 P 06d313b6f411460996d911803d6a1a0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06d313b6f411460996d911803d6a1a0e; no voters:
I20260619 09:00:17.481043 22362 leader_election.cc:288] T 00000000000000000000000000000000 P 06d313b6f411460996d911803d6a1a0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:00:17.481087 22365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06d313b6f411460996d911803d6a1a0e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:00:17.481256 22362 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06d313b6f411460996d911803d6a1a0e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:00:17.481259 22365 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06d313b6f411460996d911803d6a1a0e [term 1 LEADER]: Becoming Leader. State: Replica: 06d313b6f411460996d911803d6a1a0e, State: Running, Role: LEADER
I20260619 09:00:17.481419 22365 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06d313b6f411460996d911803d6a1a0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d313b6f411460996d911803d6a1a0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40745 } }
I20260619 09:00:17.481765 22367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06d313b6f411460996d911803d6a1a0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06d313b6f411460996d911803d6a1a0e. Latest consensus state: current_term: 1 leader_uuid: "06d313b6f411460996d911803d6a1a0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d313b6f411460996d911803d6a1a0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40745 } } }
I20260619 09:00:17.481773 22366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06d313b6f411460996d911803d6a1a0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06d313b6f411460996d911803d6a1a0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d313b6f411460996d911803d6a1a0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40745 } } }
I20260619 09:00:17.481966 22366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06d313b6f411460996d911803d6a1a0e [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:17.481889 22367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06d313b6f411460996d911803d6a1a0e [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:17.482329 22371 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:00:17.482825 22371 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:00:17.483673 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 22294
I20260619 09:00:17.483779 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 09:00:17.483958 392 external_mini_cluster.cc:1428] Setting key 106c9a8ec4f828d23d1f88a285284081
I20260619 09:00:17.484551 22371 catalog_manager.cc:1347] Generated new cluster ID: 1b0c39ca1f4b4611830fe957aa72eec3
I20260619 09:00:17.484601 22371 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:00:17.490510 22371 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:00:17.491214 22371 catalog_manager.cc:1504] Loading token signing keys...
Jun 19 09:00:17 dist-test-slave-h4g5 krb5kdc[22281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:00:17.496541 22371 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06d313b6f411460996d911803d6a1a0e: Generated new TSK 0
I20260619 09:00:17.496695 22371 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:00:17.498602 22385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:17.484408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57245 (local address 127.0.98.62:40745)
0619 09:00:17.484544 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:17.484553 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:17.490292 (+ 5739us) server_negotiation.cc:407] Connection header received
0619 09:00:17.490419 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:17.490426 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:17.490495 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:17.490627 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:17.491248 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:17.492219 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:17.492945 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:17.493361 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:17.493453 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:17.495640 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:17.495644 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:17.495644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:17.495655 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:17.497255 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:17.497562 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:17.497564 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:17.497565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:17.497647 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:17.497827 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:17.497828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:17.497828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:17.498053 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:17.498205 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:17.498272 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 09:00:17.498416 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":59,"threads_started":1}
I20260619 09:00:17.499441 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:00:17.506353 22385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:17.500761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41147 (local address 127.0.98.62:40745)
0619 09:00:17.500803 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:17.500804 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:17.500892 (+ 88us) server_negotiation.cc:407] Connection header received
0619 09:00:17.500949 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:17.500950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:17.500959 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:17.501024 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:17.501480 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:17.502203 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:17.502882 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:17.503113 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:17.503167 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:17.504188 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:17.504189 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:17.504189 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:17.504196 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:17.505401 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:17.505692 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:17.505693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:17.505693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:17.505746 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:17.505899 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:17.505900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:17.505900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:17.506011 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:17.506112 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:17.506189 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 09:00:17.506267 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 09:00:17.508044 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 22294
2026-06-19T09:00:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1209 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:00:17 dist-test-slave-h4g5 krb5kdc[22393](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:00:17 dist-test-slave-h4g5 krb5kdc[22393](info): set up 2 sockets
Jun 19 09:00:17 dist-test-slave-h4g5 krb5kdc[22393](info): commencing operation
krb5kdc: starting...
W20260619 09:00:18.455909 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 09:00:18 dist-test-slave-h4g5 krb5kdc[22393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:00:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:00:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 09:00:18.567014 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44547
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:00:18.645479 22406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:00:18.645694 22406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:00:18.645723 22406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:00:18.646893 22406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:00:18.646943 22406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:00:18.646962 22406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:00:18.646978 22406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:00:18.646992 22406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:00:18.648517 22406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:00:18.648762 22406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:00:18.648978 22406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:00:18.653674 22411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:00:18.653703 22406 server_base.cc:1034] running on GCE node
W20260619 09:00:18.653765 22412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:18.653647 22414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:00:18.654098 22406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:00:18.654371 22406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:00:18.655510 22406 hybrid_clock.cc:648] HybridClock initialized: now 1781859618655496 us; error 27 us; skew 500 ppm
Jun 19 09:00:18 dist-test-slave-h4g5 krb5kdc[22393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:00:18.662492 22406 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:00:18.662951 22406 webserver.cc:460] Webserver started at http://127.0.98.62:39525/ using document root <none> and password file <none>
I20260619 09:00:18.663151 22406 fs_manager.cc:359] Metadata directory not provided
I20260619 09:00:18.663197 22406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:00:18.663335 22406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:00:18.664471 22406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "e3f1b2d7bd7c4a1dbaa307283fa9c64c"
format_stamp: "Formatted at 2026-06-19 09:00:18 on dist-test-slave-h4g5"
server_key: "5024a1bfc78e7628bc9e7c10ba6f1f96"
server_key_iv: "07c3c7ea193aa065d4ad39c01fa28e5e"
server_key_version: "encryptionkey@0"
I20260619 09:00:18.664799 22406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "e3f1b2d7bd7c4a1dbaa307283fa9c64c"
format_stamp: "Formatted at 2026-06-19 09:00:18 on dist-test-slave-h4g5"
server_key: "5024a1bfc78e7628bc9e7c10ba6f1f96"
server_key_iv: "07c3c7ea193aa065d4ad39c01fa28e5e"
server_key_version: "encryptionkey@0"
I20260619 09:00:18.666038 22406 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:00:18.666719 22421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:00:18.666862 22406 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:00:18.666951 22406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "e3f1b2d7bd7c4a1dbaa307283fa9c64c"
format_stamp: "Formatted at 2026-06-19 09:00:18 on dist-test-slave-h4g5"
server_key: "5024a1bfc78e7628bc9e7c10ba6f1f96"
server_key_iv: "07c3c7ea193aa065d4ad39c01fa28e5e"
server_key_version: "encryptionkey@0"
I20260619 09:00:18.667009 22406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:00:18.693069 22406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:00:18.695215 22406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:00:18.695364 22406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:00:18.699136 22406 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44547
I20260619 09:00:18.699172 22473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44547 every 8 connection(s)
I20260619 09:00:18.699515 22406 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:00:18.700462 22474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:00:18.701669 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 22406
I20260619 09:00:18.701764 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 09:00:18.701924 392 external_mini_cluster.cc:1428] Setting key 7a0e8b95eda45c0296b4563a904535bc
I20260619 09:00:18.703047 22474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3f1b2d7bd7c4a1dbaa307283fa9c64c: Bootstrap starting.
I20260619 09:00:18.703825 22474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3f1b2d7bd7c4a1dbaa307283fa9c64c: Neither blocks nor log segments found. Creating new log.
I20260619 09:00:18.704151 22474 log.cc:824] T 00000000000000000000000000000000 P e3f1b2d7bd7c4a1dbaa307283fa9c64c: Log is configured to *not* fsync() on all Append() calls
I20260619 09:00:18.704950 22474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3f1b2d7bd7c4a1dbaa307283fa9c64c: No bootstrap required, opened a new log
I20260619 09:00:18.707072 22474 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3f1b2d7bd7c4a1dbaa307283fa9c64c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f1b2d7bd7c4a1dbaa307283fa9c64c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44547 } }
I20260619 09:00:18.707288 22474 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3f1b2d7bd7c4a1dbaa307283fa9c64c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:00:18.707352 22474 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3f1b2d7bd7c4a1dbaa307283fa9c64c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3f1b2d7bd7c4a1dbaa307283fa9c64c, State: Initialized, Role: FOLLOWER
I20260619 09:00:18.707499 22477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:18.702407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35747 (local address 127.0.98.62:44547)
0619 09:00:18.702581 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:18.702588 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:18.702613 (+ 25us) server_negotiation.cc:407] Connection header received
0619 09:00:18.703017 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:18.703024 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:18.703071 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:18.703201 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:18.703413 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:18.704428 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:18.705528 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:18.705972 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:18.706088 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:18.706791 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:18.706794 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:18.706795 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:00:18.706990 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:18.707022 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:00:18.707033 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:18.707082 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:18.707177 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 09:00:18.707228 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260619 09:00:18.707492 22474 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3f1b2d7bd7c4a1dbaa307283fa9c64c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f1b2d7bd7c4a1dbaa307283fa9c64c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44547 } }
I20260619 09:00:18.707578 22474 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3f1b2d7bd7c4a1dbaa307283fa9c64c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:00:18.707612 22474 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3f1b2d7bd7c4a1dbaa307283fa9c64c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:00:18.707652 22474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3f1b2d7bd7c4a1dbaa307283fa9c64c [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:00:18.708287 22474 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3f1b2d7bd7c4a1dbaa307283fa9c64c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f1b2d7bd7c4a1dbaa307283fa9c64c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44547 } }
I20260619 09:00:18.708354 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:00:18.708388 22474 leader_election.cc:302] T 00000000000000000000000000000000 P e3f1b2d7bd7c4a1dbaa307283fa9c64c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3f1b2d7bd7c4a1dbaa307283fa9c64c; no voters:
I20260619 09:00:18.708545 22474 leader_election.cc:288] T 00000000000000000000000000000000 P e3f1b2d7bd7c4a1dbaa307283fa9c64c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:00:18.708583 22479 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3f1b2d7bd7c4a1dbaa307283fa9c64c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:00:18.708734 22479 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3f1b2d7bd7c4a1dbaa307283fa9c64c [term 1 LEADER]: Becoming Leader. State: Replica: e3f1b2d7bd7c4a1dbaa307283fa9c64c, State: Running, Role: LEADER
I20260619 09:00:18.708774 22474 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3f1b2d7bd7c4a1dbaa307283fa9c64c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:00:18.708926 22479 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3f1b2d7bd7c4a1dbaa307283fa9c64c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f1b2d7bd7c4a1dbaa307283fa9c64c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44547 } }
I20260619 09:00:18.709302 22481 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3f1b2d7bd7c4a1dbaa307283fa9c64c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3f1b2d7bd7c4a1dbaa307283fa9c64c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f1b2d7bd7c4a1dbaa307283fa9c64c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44547 } } }
I20260619 09:00:18.709375 22481 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3f1b2d7bd7c4a1dbaa307283fa9c64c [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:18.709995 22490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:00:18.710197 22480 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3f1b2d7bd7c4a1dbaa307283fa9c64c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3f1b2d7bd7c4a1dbaa307283fa9c64c. Latest consensus state: current_term: 1 leader_uuid: "e3f1b2d7bd7c4a1dbaa307283fa9c64c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f1b2d7bd7c4a1dbaa307283fa9c64c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44547 } } }
I20260619 09:00:18.710405 22480 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3f1b2d7bd7c4a1dbaa307283fa9c64c [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:18.710494 22490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:00:18.712168 22490 catalog_manager.cc:1347] Generated new cluster ID: e87ea71170db4f28b7a12b5be8cb61fb
I20260619 09:00:18.712222 22490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:00:18.713076 22477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:18.709832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52437 (local address 127.0.98.62:44547)
0619 09:00:18.709883 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:18.709885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:18.709934 (+ 49us) server_negotiation.cc:407] Connection header received
0619 09:00:18.710039 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:18.710040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:18.710050 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:18.710113 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:18.710327 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:18.710984 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:18.711820 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:18.712039 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:18.712129 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:18.712835 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:18.712836 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:18.712837 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:00:18.712842 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:18.712857 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:00:18.712862 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:18.712934 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:18.712986 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 09:00:18.713008 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 09:00:18.721714 22490 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:00:18.722272 22490 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:00:18.729905 22490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3f1b2d7bd7c4a1dbaa307283fa9c64c: Generated new TSK 0
I20260619 09:00:18.730114 22490 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:00:18.732620 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 22406
2026-06-19T09:00:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1224 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:00:18 dist-test-slave-h4g5 krb5kdc[22505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:00:18 dist-test-slave-h4g5 krb5kdc[22505](info): set up 2 sockets
Jun 19 09:00:18 dist-test-slave-h4g5 krb5kdc[22505](info): commencing operation
krb5kdc: starting...
W20260619 09:00:19.679875 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 09:00:19 dist-test-slave-h4g5 krb5kdc[22505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:00:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:00:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 09:00:19.792891 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40623
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:00:19.872270 22518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:00:19.872484 22518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:00:19.872507 22518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:00:19.873632 22518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:00:19.873669 22518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:00:19.873682 22518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:00:19.873692 22518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:00:19.873703 22518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:00:19.875123 22518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:00:19.875312 22518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:00:19.875495 22518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:00:19.880185 22526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:19.880188 22524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:00:19.880188 22518 server_base.cc:1034] running on GCE node
W20260619 09:00:19.880333 22523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:00:19.880748 22518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:00:19.881049 22518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:00:19.882193 22518 hybrid_clock.cc:648] HybridClock initialized: now 1781859619882175 us; error 34 us; skew 500 ppm
Jun 19 09:00:19 dist-test-slave-h4g5 krb5kdc[22505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:00:19.884649 22518 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:00:19.885068 22518 webserver.cc:460] Webserver started at http://127.0.98.62:41891/ using document root <none> and password file <none>
I20260619 09:00:19.885284 22518 fs_manager.cc:359] Metadata directory not provided
I20260619 09:00:19.885329 22518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:00:19.885447 22518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:00:19.886482 22518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "a6c9c1765fe644a0b5ff8d97a30e131a"
format_stamp: "Formatted at 2026-06-19 09:00:19 on dist-test-slave-h4g5"
server_key: "08f3e1420754870a4ab54a87c95c7ed9"
server_key_iv: "95b657e9bb82f2ba67afad4584998044"
server_key_version: "encryptionkey@0"
I20260619 09:00:19.886801 22518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "a6c9c1765fe644a0b5ff8d97a30e131a"
format_stamp: "Formatted at 2026-06-19 09:00:19 on dist-test-slave-h4g5"
server_key: "08f3e1420754870a4ab54a87c95c7ed9"
server_key_iv: "95b657e9bb82f2ba67afad4584998044"
server_key_version: "encryptionkey@0"
I20260619 09:00:19.888029 22518 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:00:19.888779 22533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:00:19.888921 22518 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:00:19.889004 22518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "a6c9c1765fe644a0b5ff8d97a30e131a"
format_stamp: "Formatted at 2026-06-19 09:00:19 on dist-test-slave-h4g5"
server_key: "08f3e1420754870a4ab54a87c95c7ed9"
server_key_iv: "95b657e9bb82f2ba67afad4584998044"
server_key_version: "encryptionkey@0"
I20260619 09:00:19.889075 22518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:00:19.892289 22518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:00:19.893991 22518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:00:19.894116 22518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:00:19.897907 22518 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40623
I20260619 09:00:19.897950 22585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40623 every 8 connection(s)
I20260619 09:00:19.898249 22518 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:00:19.899170 22586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:00:19.901441 22586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6c9c1765fe644a0b5ff8d97a30e131a: Bootstrap starting.
I20260619 09:00:19.901970 22586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6c9c1765fe644a0b5ff8d97a30e131a: Neither blocks nor log segments found. Creating new log.
I20260619 09:00:19.902190 22586 log.cc:824] T 00000000000000000000000000000000 P a6c9c1765fe644a0b5ff8d97a30e131a: Log is configured to *not* fsync() on all Append() calls
I20260619 09:00:19.902819 22586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6c9c1765fe644a0b5ff8d97a30e131a: No bootstrap required, opened a new log
I20260619 09:00:19.904450 22586 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6c9c1765fe644a0b5ff8d97a30e131a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c9c1765fe644a0b5ff8d97a30e131a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40623 } }
I20260619 09:00:19.904594 22586 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6c9c1765fe644a0b5ff8d97a30e131a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:00:19.904615 22586 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6c9c1765fe644a0b5ff8d97a30e131a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6c9c1765fe644a0b5ff8d97a30e131a, State: Initialized, Role: FOLLOWER
I20260619 09:00:19.904729 22586 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6c9c1765fe644a0b5ff8d97a30e131a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c9c1765fe644a0b5ff8d97a30e131a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40623 } }
I20260619 09:00:19.904803 22586 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6c9c1765fe644a0b5ff8d97a30e131a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:00:19.904831 22586 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6c9c1765fe644a0b5ff8d97a30e131a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:00:19.904861 22586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6c9c1765fe644a0b5ff8d97a30e131a [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:00:19.905382 22586 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6c9c1765fe644a0b5ff8d97a30e131a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c9c1765fe644a0b5ff8d97a30e131a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40623 } }
I20260619 09:00:19.905470 22586 leader_election.cc:302] T 00000000000000000000000000000000 P a6c9c1765fe644a0b5ff8d97a30e131a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6c9c1765fe644a0b5ff8d97a30e131a; no voters:
I20260619 09:00:19.905596 22586 leader_election.cc:288] T 00000000000000000000000000000000 P a6c9c1765fe644a0b5ff8d97a30e131a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:00:19.905652 22589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6c9c1765fe644a0b5ff8d97a30e131a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:00:19.905835 22586 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6c9c1765fe644a0b5ff8d97a30e131a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:00:19.905821 22589 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6c9c1765fe644a0b5ff8d97a30e131a [term 1 LEADER]: Becoming Leader. State: Replica: a6c9c1765fe644a0b5ff8d97a30e131a, State: Running, Role: LEADER
I20260619 09:00:19.905967 22589 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6c9c1765fe644a0b5ff8d97a30e131a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c9c1765fe644a0b5ff8d97a30e131a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40623 } }
I20260619 09:00:19.906510 22591 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6c9c1765fe644a0b5ff8d97a30e131a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6c9c1765fe644a0b5ff8d97a30e131a. Latest consensus state: current_term: 1 leader_uuid: "a6c9c1765fe644a0b5ff8d97a30e131a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c9c1765fe644a0b5ff8d97a30e131a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40623 } } }
I20260619 09:00:19.906526 22590 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6c9c1765fe644a0b5ff8d97a30e131a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6c9c1765fe644a0b5ff8d97a30e131a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6c9c1765fe644a0b5ff8d97a30e131a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40623 } } }
I20260619 09:00:19.906684 22590 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6c9c1765fe644a0b5ff8d97a30e131a [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:19.906903 22598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:00:19.906672 22591 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6c9c1765fe644a0b5ff8d97a30e131a [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:19.907514 22598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:00:19.907675 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 22518
I20260619 09:00:19.907773 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 09:00:19.907968 392 external_mini_cluster.cc:1428] Setting key 22d9cb682d7ead20609f60ade37654f3
I20260619 09:00:19.909391 22598 catalog_manager.cc:1347] Generated new cluster ID: 41d25562962841108301682b63c7149d
I20260619 09:00:19.909446 22598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:00:19.913761 22608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:19.908617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54935 (local address 127.0.98.62:40623)
0619 09:00:19.908786 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:19.908795 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:19.912416 (+ 3621us) server_negotiation.cc:407] Connection header received
0619 09:00:19.912528 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:19.912534 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:19.912560 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:19.912672 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:19.913348 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:19.913352 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:19.913352 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:00:19.913357 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:19.913391 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:00:19.913402 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:19.913457 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:19.913526 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 09:00:19.913569 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":75,"threads_started":1}
I20260619 09:00:19.914572 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:00:19.916828 22608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:19.915844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34815 (local address 127.0.98.62:40623)
0619 09:00:19.915908 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:19.915910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:19.916005 (+ 95us) server_negotiation.cc:407] Connection header received
0619 09:00:19.916057 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:19.916058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:19.916063 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:19.916123 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:19.916644 (+ 521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:19.916645 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:19.916645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:00:19.916649 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:19.916662 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:00:19.916665 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:19.916711 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:19.916757 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 09:00:19.916768 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 09:00:19.944339 22598 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:00:19.944542 22598 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:00:19.955760 22598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6c9c1765fe644a0b5ff8d97a30e131a: Generated new TSK 0
I20260619 09:00:19.955976 22598 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:00:19.959911 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 22518
2026-06-19T09:00:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1227 ms)
[----------] 18 tests from AuthTokenIssuingTest (21925 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:00:20 dist-test-slave-h4g5 krb5kdc[22617](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:00:20 dist-test-slave-h4g5 krb5kdc[22617](info): set up 2 sockets
Jun 19 09:00:20 dist-test-slave-h4g5 krb5kdc[22617](info): commencing operation
krb5kdc: starting...
W20260619 09:00:20.915993 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 09:00:20 dist-test-slave-h4g5 krb5kdc[22617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:00:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:00:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 09:00:21.028884 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41553
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41553,127.0.98.61:40117,127.0.98.60:33923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:00:21.114135 22630 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 09:00:21.114465 22630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:00:21.114507 22630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:00:21.114594 22630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:00:21.116420 22630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:00:21.116470 22630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:00:21.116484 22630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:00:21.116495 22630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:00:21.116506 22630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:00:21.118827 22630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41553,127.0.98.61:40117,127.0.98.60:33923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:00:21.119138 22630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:00:21.119402 22630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:00:21.124521 22636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:21.124529 22638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:21.124529 22635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:00:21.124784 22630 server_base.cc:1034] running on GCE node
I20260619 09:00:21.125010 22630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:00:21.125279 22630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:00:21.126415 22630 hybrid_clock.cc:648] HybridClock initialized: now 1781859621126392 us; error 38 us; skew 500 ppm
Jun 19 09:00:21 dist-test-slave-h4g5 krb5kdc[22617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:00:21.129293 22630 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:00:21.129738 22630 webserver.cc:460] Webserver started at http://127.0.98.62:41915/ using document root <none> and password file <none>
I20260619 09:00:21.129951 22630 fs_manager.cc:359] Metadata directory not provided
I20260619 09:00:21.129998 22630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:00:21.130123 22630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:00:21.131197 22630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "79b42efdeaf244298b61f2e08c1273b2"
format_stamp: "Formatted at 2026-06-19 09:00:21 on dist-test-slave-h4g5"
server_key: "ae6a727040d52270d7750f0d44c498d2"
server_key_iv: "5e977315633eb176e51aa6ee12cec9b1"
server_key_version: "encryptionkey@0"
I20260619 09:00:21.131526 22630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "79b42efdeaf244298b61f2e08c1273b2"
format_stamp: "Formatted at 2026-06-19 09:00:21 on dist-test-slave-h4g5"
server_key: "ae6a727040d52270d7750f0d44c498d2"
server_key_iv: "5e977315633eb176e51aa6ee12cec9b1"
server_key_version: "encryptionkey@0"
I20260619 09:00:21.132771 22630 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 09:00:21.133476 22645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:00:21.133625 22630 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:00:21.133703 22630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "79b42efdeaf244298b61f2e08c1273b2"
format_stamp: "Formatted at 2026-06-19 09:00:21 on dist-test-slave-h4g5"
server_key: "ae6a727040d52270d7750f0d44c498d2"
server_key_iv: "5e977315633eb176e51aa6ee12cec9b1"
server_key_version: "encryptionkey@0"
I20260619 09:00:21.133765 22630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:00:21.149618 22630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:00:21.151250 22630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:00:21.151383 22630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:00:21.155011 22630 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41553
I20260619 09:00:21.155054 22697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41553 every 8 connection(s)
I20260619 09:00:21.155371 22630 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:00:21.156309 22698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:00:21.157799 22698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41553 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 09:00:21 dist-test-slave-h4g5 krb5kdc[22617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:00:21.163542 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 22630
I20260619 09:00:21.163630 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 09:00:21.163827 392 external_mini_cluster.cc:1428] Setting key 8440585a6aff085afd5f25276eeeb2f8
I20260619 09:00:21.166944 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.158421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52326 (local address 127.0.98.62:41553)
0619 09:00:21.158550 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:21.158560 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:21.158667 (+ 107us) server_negotiation.cc:407] Connection header received
0619 09:00:21.159216 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:21.159219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:21.159243 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:21.159318 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:21.159838 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.160684 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.161511 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.161807 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.161901 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:21.163788 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:21.163797 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:21.163798 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:21.163805 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:21.165351 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:21.165784 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:21.165787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:21.165788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:21.165874 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:21.166112 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:21.166113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:21.166113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:21.166324 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:21.166388 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:21.166616 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 09:00:21.166776 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":76,"server-negotiator.queue_time_us":98,"thread_start_us":65,"threads_started":1}
I20260619 09:00:21.166949 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.158376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41553 (local address 127.0.0.1:52326)
0619 09:00:21.158540 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:21.158581 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 09:00:21.159072 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:00:21.159330 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:00:21.159332 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:00:21.159592 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:00:21.159751 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:21.159756 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.160769 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.160770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:21.161436 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:21.161438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.161893 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.161894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:21.161955 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:21.162385 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:00:21.162393 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:00:21.163711 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:00:21.165425 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:21.165431 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:21.165434 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:21.165677 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:21.165943 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:21.165944 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:21.165944 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:21.166051 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:21.166386 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:00:21.166393 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:00:21.166466 (+ 73us) client_negotiation.cc:769] Sending connection context
0619 09:00:21.166561 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 09:00:21.166725 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
I20260619 09:00:21.167695 22698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40117 } has no permanent_uuid. Determining permanent_uuid...
I20260619 09:00:21.168020 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.167904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40117 (local address 127.0.0.1:46350)
0619 09:00:21.167949 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:21.167995 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 09:00:21.168176 22649 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40117: connect: Connection refused (error 111)
I20260619 09:00:21.168594 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.168500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40117 (local address 127.0.0.1:46360)
0619 09:00:21.168543 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:21.168571 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 09:00:21.168689 22698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40117: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40117: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 09:00:21.188927 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40117
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41553,127.0.98.61:40117,127.0.98.60:33923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 09:00:21.217922 22698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40117 } attempt: 1
I20260619 09:00:21.218489 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.218323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40117 (local address 127.0.0.1:46366)
0619 09:00:21.218391 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:21.218451 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 09:00:21.218842 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.218751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40117 (local address 127.0.0.1:46372)
0619 09:00:21.218790 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:21.218820 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 09:00:21.218947 22698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40117: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40117: connect: Connection refused (error 111)
W20260619 09:00:21.269546 22705 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 09:00:21.269758 22705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:00:21.269775 22705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:00:21.269793 22705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:00:21.270879 22705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:00:21.270910 22705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:00:21.270922 22705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:00:21.270933 22705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:00:21.270943 22705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:00:21.272447 22705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41553,127.0.98.61:40117,127.0.98.60:33923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:00:21.272665 22705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:00:21.272847 22705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:00:21.277652 22710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:21.277696 22713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:21.277683 22711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:00:21.277796 22705 server_base.cc:1034] running on GCE node
I20260619 09:00:21.278064 22705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:00:21.278349 22705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:00:21.279480 22705 hybrid_clock.cc:648] HybridClock initialized: now 1781859621279475 us; error 32 us; skew 500 ppm
Jun 19 09:00:21 dist-test-slave-h4g5 krb5kdc[22617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:00:21.281814 22705 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 09:00:21.282229 22705 webserver.cc:460] Webserver started at http://127.0.98.61:38625/ using document root <none> and password file <none>
I20260619 09:00:21.282434 22705 fs_manager.cc:359] Metadata directory not provided
I20260619 09:00:21.282481 22705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:00:21.282600 22705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:00:21.283595 22705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-1/data/instance:
uuid: "adcdc77475674133ac1b3086d879e85d"
format_stamp: "Formatted at 2026-06-19 09:00:21 on dist-test-slave-h4g5"
server_key: "871d7bd5024762f468bcc20ef39e05a0"
server_key_iv: "e7125416916db20e2c1a5fbf1452d3b0"
server_key_version: "encryptionkey@0"
I20260619 09:00:21.283915 22705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-1/wal/instance:
uuid: "adcdc77475674133ac1b3086d879e85d"
format_stamp: "Formatted at 2026-06-19 09:00:21 on dist-test-slave-h4g5"
server_key: "871d7bd5024762f468bcc20ef39e05a0"
server_key_iv: "e7125416916db20e2c1a5fbf1452d3b0"
server_key_version: "encryptionkey@0"
I20260619 09:00:21.285132 22705 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:00:21.285800 22720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:00:21.285961 22705 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:00:21.286039 22705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-1/wal
uuid: "adcdc77475674133ac1b3086d879e85d"
format_stamp: "Formatted at 2026-06-19 09:00:21 on dist-test-slave-h4g5"
server_key: "871d7bd5024762f468bcc20ef39e05a0"
server_key_iv: "e7125416916db20e2c1a5fbf1452d3b0"
server_key_version: "encryptionkey@0"
I20260619 09:00:21.286103 22705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:00:21.287086 22698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40117 } attempt: 2
I20260619 09:00:21.287469 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.287349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40117 (local address 127.0.0.1:46388)
0619 09:00:21.287403 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:21.287439 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 09:00:21.287823 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.287737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40117 (local address 127.0.0.1:46398)
0619 09:00:21.287775 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:21.287801 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 09:00:21.287949 22698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40117: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40117: connect: Connection refused (error 111)
I20260619 09:00:21.306674 22705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:00:21.308663 22705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:00:21.308797 22705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:00:21.312732 22705 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40117
I20260619 09:00:21.312847 22772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40117 every 8 connection(s)
I20260619 09:00:21.313172 22705 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-1/data/info.pb
I20260619 09:00:21.313524 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 22705
I20260619 09:00:21.313621 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-1/wal/instance
I20260619 09:00:21.313814 392 external_mini_cluster.cc:1428] Setting key ad3751ff286d48de4296e824d9b42f8a
I20260619 09:00:21.314371 22773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:00:21.316427 22773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41553 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 19 09:00:21 dist-test-slave-h4g5 krb5kdc[22617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:00:21.328133 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.317597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52332 (local address 127.0.98.62:41553)
0619 09:00:21.317648 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:21.317650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:21.317663 (+ 13us) server_negotiation.cc:407] Connection header received
0619 09:00:21.318048 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:21.318050 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:21.318066 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:21.318127 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:21.319100 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.319840 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.321386 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.321581 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.321638 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:21.325195 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:21.325197 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:21.325197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:21.325205 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:21.326661 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:21.327135 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:21.327136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:21.327137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:21.327202 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:21.327460 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:21.327461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:21.327461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:21.327580 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:21.327651 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:21.327955 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 09:00:21.328028 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 09:00:21.328398 22776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.317143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41553 (local address 127.0.0.1:52332)
0619 09:00:21.317327 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:21.317376 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 09:00:21.317951 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:00:21.318270 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:00:21.318275 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:00:21.318668 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:00:21.318979 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:21.318985 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.319953 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.319954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:21.321307 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:21.321309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.321685 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.321686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:21.321823 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:21.322310 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:00:21.322319 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:00:21.325099 (+ 2780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:00:21.326745 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:21.326753 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:21.326756 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:21.327046 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:21.327262 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:21.327263 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:21.327263 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:21.327392 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:21.327638 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:00:21.327644 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:00:21.327815 (+ 171us) client_negotiation.cc:769] Sending connection context
0619 09:00:21.327938 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 09:00:21.328135 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
I20260619 09:00:21.328969 22773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40117 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 09:00:21 dist-test-slave-h4g5 krb5kdc[22617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 09:00:21.337270 22776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.329328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40117 (local address 127.0.0.1:46400)
0619 09:00:21.329378 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:21.329388 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:00:21.329422 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:00:21.329667 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:00:21.329668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:00:21.329808 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:00:21.329870 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:21.329872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.330699 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.330700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:21.331490 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:21.331493 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.331976 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.331978 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:21.332058 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:21.332473 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:00:21.332478 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:00:21.334618 (+ 2140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:00:21.336358 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:21.336359 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:21.336360 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:21.336509 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:21.336669 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:21.336670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:21.336670 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:21.336718 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:21.336988 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:00:21.336989 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:00:21.337048 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 09:00:21.337114 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 09:00:21.337187 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 09:00:21.337375 22778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.329389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46400 (local address 127.0.98.61:40117)
0619 09:00:21.329531 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:21.329537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:21.329556 (+ 19us) server_negotiation.cc:407] Connection header received
0619 09:00:21.329575 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:21.329576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:21.329587 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:21.329649 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:21.329933 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.330621 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.331576 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.331907 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.331958 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:21.334723 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:21.334734 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:21.334734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:21.334740 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:21.336281 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:21.336570 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:21.336572 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:21.336573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:21.336622 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:21.336758 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:21.336759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:21.336759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:21.336939 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:21.337002 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:21.337162 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 09:00:21.337250 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":115,"thread_start_us":78,"threads_started":1}
I20260619 09:00:21.337889 22773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33923 } has no permanent_uuid. Determining permanent_uuid...
I20260619 09:00:21.338201 22776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.338112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33923 (local address 127.0.0.1:54880)
0619 09:00:21.338142 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:21.338185 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260619 09:00:21.338313 22724 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33923: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 09:00:21.338711 22776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.338624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33923 (local address 127.0.0.1:54896)
0619 09:00:21.338663 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:21.338691 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260619 09:00:21.338806 22773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33923: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33923: connect: Connection refused (error 111)
I20260619 09:00:21.339156 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33923
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41553,127.0.98.61:40117,127.0.98.60:33923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 09:00:21.379103 22698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40117 } attempt: 3
Jun 19 09:00:21 dist-test-slave-h4g5 krb5kdc[22617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 09:00:21.385475 22778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.379537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46416 (local address 127.0.98.61:40117)
0619 09:00:21.379598 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:21.379600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:21.379622 (+ 22us) server_negotiation.cc:407] Connection header received
0619 09:00:21.379648 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:21.379649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:21.379660 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:21.379702 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:21.380003 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.380535 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.381266 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.381482 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.381538 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:21.383278 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:21.383279 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:21.383280 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:21.383287 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:21.384484 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:21.384776 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:21.384777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:21.384777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:21.384827 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:21.384992 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:21.384993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:21.384993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:21.385107 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:21.385170 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:21.385301 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 09:00:21.385370 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 09:00:21.385475 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.379457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40117 (local address 127.0.0.1:46416)
0619 09:00:21.379521 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:21.379532 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:00:21.379578 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:00:21.379714 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:00:21.379715 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:00:21.379867 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:00:21.379933 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:21.379934 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.380656 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.380657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:21.381185 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:21.381187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.381541 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.381541 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:21.381624 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:21.382080 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:00:21.382084 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:00:21.383204 (+ 1120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:00:21.384554 (+ 1350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:21.384555 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:21.384555 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:21.384717 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:21.384889 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:21.384890 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:21.384890 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:21.384945 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:21.385166 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:00:21.385167 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:00:21.385214 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 09:00:21.385273 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 09:00:21.385370 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 09:00:21.385846 22698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33923 } has no permanent_uuid. Determining permanent_uuid...
I20260619 09:00:21.386150 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.386060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33923 (local address 127.0.0.1:54900)
0619 09:00:21.386100 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:21.386126 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260619 09:00:21.386454 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.386384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33923 (local address 127.0.0.1:54906)
0619 09:00:21.386420 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:21.386440 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 09:00:21.386526 22698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33923: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33923: connect: Connection refused (error 111)
I20260619 09:00:21.388002 22773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33923 } attempt: 1
I20260619 09:00:21.388315 22776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.388219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33923 (local address 127.0.0.1:54920)
0619 09:00:21.388263 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:21.388288 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 09:00:21.388602 22776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.388544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33923 (local address 127.0.0.1:54936)
0619 09:00:21.388567 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:21.388587 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260619 09:00:21.388692 22773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33923: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33923: connect: Connection refused (error 111)
I20260619 09:00:21.417657 22698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33923 } attempt: 1
I20260619 09:00:21.418146 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.418019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33923 (local address 127.0.0.1:54950)
0619 09:00:21.418075 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:21.418119 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 09:00:21.418476 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.418394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33923 (local address 127.0.0.1:54964)
0619 09:00:21.418434 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:21.418457 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 09:00:21.418574 22698 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33923: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33923: connect: Connection refused (error 111)
W20260619 09:00:21.418872 22780 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 09:00:21.419067 22780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:00:21.419088 22780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:00:21.419111 22780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:00:21.420233 22780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:00:21.420269 22780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:00:21.420281 22780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:00:21.420292 22780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:00:21.420308 22780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:00:21.421676 22780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41553,127.0.98.61:40117,127.0.98.60:33923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:00:21.421855 22780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:00:21.422024 22780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:00:21.426497 22788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:21.426559 22786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:21.426504 22785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:00:21.426733 22780 server_base.cc:1034] running on GCE node
I20260619 09:00:21.426934 22780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:00:21.427162 22780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:00:21.428299 22780 hybrid_clock.cc:648] HybridClock initialized: now 1781859621428286 us; error 26 us; skew 500 ppm
Jun 19 09:00:21 dist-test-slave-h4g5 krb5kdc[22617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:00:21.430789 22780 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 09:00:21.431217 22780 webserver.cc:460] Webserver started at http://127.0.98.60:45451/ using document root <none> and password file <none>
I20260619 09:00:21.431409 22780 fs_manager.cc:359] Metadata directory not provided
I20260619 09:00:21.431452 22780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:00:21.431592 22780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:00:21.432705 22780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-2/data/instance:
uuid: "15383ec9a2624d08974db8ee13127745"
format_stamp: "Formatted at 2026-06-19 09:00:21 on dist-test-slave-h4g5"
server_key: "f37aa7f66a97102a56e0d06ae477ed33"
server_key_iv: "930802724e7d68247d4d3f3553e2acb4"
server_key_version: "encryptionkey@0"
I20260619 09:00:21.433054 22780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-2/wal/instance:
uuid: "15383ec9a2624d08974db8ee13127745"
format_stamp: "Formatted at 2026-06-19 09:00:21 on dist-test-slave-h4g5"
server_key: "f37aa7f66a97102a56e0d06ae477ed33"
server_key_iv: "930802724e7d68247d4d3f3553e2acb4"
server_key_version: "encryptionkey@0"
I20260619 09:00:21.434342 22780 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:00:21.435082 22795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:00:21.435246 22780 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:00:21.435328 22780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-2/wal
uuid: "15383ec9a2624d08974db8ee13127745"
format_stamp: "Formatted at 2026-06-19 09:00:21 on dist-test-slave-h4g5"
server_key: "f37aa7f66a97102a56e0d06ae477ed33"
server_key_iv: "930802724e7d68247d4d3f3553e2acb4"
server_key_version: "encryptionkey@0"
I20260619 09:00:21.435398 22780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:00:21.456836 22773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33923 } attempt: 2
I20260619 09:00:21.457309 22776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.457185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33923 (local address 127.0.0.1:54972)
0619 09:00:21.457241 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:21.457269 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 09:00:21.457621 22776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.457553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33923 (local address 127.0.0.1:54986)
0619 09:00:21.457590 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:21.457605 (+ 15us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 09:00:21.457696 22773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33923: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33923: connect: Connection refused (error 111)
I20260619 09:00:21.463150 22780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:00:21.465135 22780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:00:21.465238 22780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:00:21.468888 22780 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:33923
I20260619 09:00:21.468920 22847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:33923 every 8 connection(s)
I20260619 09:00:21.469236 22780 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-2/data/info.pb
I20260619 09:00:21.470126 22848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:00:21.471654 22848 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41553 } has no permanent_uuid. Determining permanent_uuid...
I20260619 09:00:21.473702 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 22780
I20260619 09:00:21.473778 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859361540516-392-0/minicluster-data/master-2/wal/instance
I20260619 09:00:21.473949 392 external_mini_cluster.cc:1428] Setting key d9508ddc40bd3a007ccafa40ce5dc719
Jun 19 09:00:21 dist-test-slave-h4g5 krb5kdc[22617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 09:00:21 dist-test-slave-h4g5 krb5kdc[22617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 09:00:21.480309 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.472342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52340 (local address 127.0.98.62:41553)
0619 09:00:21.472400 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:21.472401 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:21.472548 (+ 147us) server_negotiation.cc:407] Connection header received
0619 09:00:21.473003 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:21.473004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:21.473014 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:21.473070 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:21.473729 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.474414 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.475388 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.475589 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.475642 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:21.477711 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:21.477712 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:21.477713 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:21.477721 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:21.479026 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:21.479379 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:21.479380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:21.479380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:21.479435 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:21.479691 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:21.479692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:21.479693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:21.479803 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:21.479857 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:21.480138 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 09:00:21.480219 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 09:00:21.480453 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.472300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41553 (local address 127.0.0.1:52340)
0619 09:00:21.472435 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:21.472476 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 09:00:21.472923 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:00:21.473122 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:00:21.473125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:00:21.473354 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:00:21.473553 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:21.473559 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.474509 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.474510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:21.475301 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:21.475303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.475672 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.475673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:21.475780 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:21.476286 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:00:21.476294 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:00:21.477630 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:00:21.479099 (+ 1469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:21.479104 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:21.479106 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:21.479308 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:21.479530 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:21.479531 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:21.479531 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:21.479633 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:21.479857 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:00:21.479863 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:00:21.479982 (+ 119us) client_negotiation.cc:769] Sending connection context
0619 09:00:21.480072 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 09:00:21.480241 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":1,"thread_start_us":68,"threads_started":1}
I20260619 09:00:21.481026 22848 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40117 } has no permanent_uuid. Determining permanent_uuid...
I20260619 09:00:21.482013 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.474282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54990 (local address 127.0.98.60:33923)
0619 09:00:21.474431 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:21.474436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:21.474508 (+ 72us) server_negotiation.cc:407] Connection header received
0619 09:00:21.474552 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:21.474554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:21.474573 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:21.474638 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:21.475175 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.475864 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.476602 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.476901 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.476957 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:21.479291 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:21.479298 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:21.479298 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:21.479305 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:21.480943 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:21.481246 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:21.481249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:21.481249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:21.481303 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:21.481491 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:21.481492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:21.481492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:21.481646 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:21.481795 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:21.481838 (+ 43us) server_negotiation.cc:299] Negotiation successful
0619 09:00:21.481912 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
Jun 19 09:00:21 dist-test-slave-h4g5 krb5kdc[22617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 09:00:21 dist-test-slave-h4g5 krb5kdc[22617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:00:21.487185 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.481267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40117 (local address 127.0.0.1:46418)
0619 09:00:21.481302 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:21.481313 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:00:21.481344 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:00:21.481470 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:00:21.481470 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:00:21.481584 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:00:21.481613 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:21.481614 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.482375 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.482376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:21.482950 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:21.482952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.483367 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.483368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:21.483430 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:21.483795 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:00:21.483800 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:00:21.484915 (+ 1115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:00:21.486219 (+ 1304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:21.486219 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:21.486220 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:21.486404 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:21.486626 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:21.486626 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:21.486627 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:21.486684 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:21.486930 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:00:21.486931 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:00:21.486978 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 09:00:21.487040 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 09:00:21.487123 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260619 09:00:21.487238 22778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.481331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46418 (local address 127.0.98.61:40117)
0619 09:00:21.481372 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:21.481374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:21.481382 (+ 8us) server_negotiation.cc:407] Connection header received
0619 09:00:21.481404 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:21.481405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:21.481414 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:21.481457 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:21.481680 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.482308 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.483043 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.483307 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.483363 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:21.484984 (+ 1621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:21.484985 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:21.484985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:21.484991 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:21.486147 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:21.486475 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:21.486476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:21.486477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:21.486539 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:21.486746 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:21.486747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:21.486748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:21.486873 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:21.486932 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:21.487068 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 09:00:21.487148 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 09:00:21.487522 22848 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33923 } has no permanent_uuid. Determining permanent_uuid...
I20260619 09:00:21.489936 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.482688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52342 (local address 127.0.98.62:41553)
0619 09:00:21.482725 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:21.482727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:21.483078 (+ 351us) server_negotiation.cc:407] Connection header received
0619 09:00:21.483109 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:21.483110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:21.483119 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:21.483175 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:21.483546 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.483983 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.484647 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.484913 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.484980 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:21.487379 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:21.487380 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:21.487381 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:21.487387 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:21.488837 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:21.489190 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:21.489191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:21.489191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:21.489245 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:21.489410 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:21.489411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:21.489411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:21.489539 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:21.489743 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:21.489781 (+ 38us) server_negotiation.cc:299] Negotiation successful
0619 09:00:21.489851 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Jun 19 09:00:21 dist-test-slave-h4g5 krb5kdc[22617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 09:00:21.493706 22698 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33923 } attempt: 2
I20260619 09:00:21.493793 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.487754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33923 (local address 127.0.0.1:54996)
0619 09:00:21.487792 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:21.487800 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 09:00:21.487829 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:00:21.487923 (+ 94us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:00:21.487924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:00:21.488035 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:00:21.488057 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:21.488058 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.488636 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.488637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:21.489266 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:21.489268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.489714 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.489714 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:21.489758 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:21.490159 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:00:21.490164 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:00:21.491273 (+ 1109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:00:21.492717 (+ 1444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:21.492718 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:21.492718 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:21.492908 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:21.493233 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:21.493234 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:21.493234 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:21.493286 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:21.493522 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:00:21.493523 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:00:21.493554 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 09:00:21.493603 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 09:00:21.493665 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 09:00:21.494194 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.487795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54996 (local address 127.0.98.60:33923)
0619 09:00:21.487827 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:21.487828 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:21.487835 (+ 7us) server_negotiation.cc:407] Connection header received
0619 09:00:21.487871 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:21.487872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:21.487881 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:21.487922 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:21.488126 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.488577 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.489332 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.489596 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.489673 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:21.491341 (+ 1668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:21.491342 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:21.491342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:21.491348 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:21.492648 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:21.492970 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:21.492971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:21.492971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:21.493029 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:21.493341 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:21.493342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:21.493343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:21.493463 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:21.493533 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:21.493646 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 09:00:21.493713 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Jun 19 09:00:21 dist-test-slave-h4g5 krb5kdc[22617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 09:00:21.495982 22848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15383ec9a2624d08974db8ee13127745: Bootstrap starting.
I20260619 09:00:21.496783 22848 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15383ec9a2624d08974db8ee13127745: Neither blocks nor log segments found. Creating new log.
I20260619 09:00:21.497082 22848 log.cc:824] T 00000000000000000000000000000000 P 15383ec9a2624d08974db8ee13127745: Log is configured to *not* fsync() on all Append() calls
Jun 19 09:00:21 dist-test-slave-h4g5 krb5kdc[22617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 09:00:21.497857 22848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15383ec9a2624d08974db8ee13127745: No bootstrap required, opened a new log
I20260619 09:00:21.498175 22778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.490523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46430 (local address 127.0.98.61:40117)
0619 09:00:21.490565 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:21.490567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:21.490574 (+ 7us) server_negotiation.cc:407] Connection header received
0619 09:00:21.490591 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:21.490592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:21.490600 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:21.490656 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:21.491104 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.491710 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.492610 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.492842 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.492912 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:21.495453 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:21.495455 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:21.495455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:21.495461 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:21.496901 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:21.497252 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:21.497253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:21.497253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:21.497312 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:21.497482 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:21.497483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:21.497483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:21.497622 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:21.497764 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:21.498008 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 09:00:21.498079 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 09:00:21.499562 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:00:21.499779 22848 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15383ec9a2624d08974db8ee13127745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b42efdeaf244298b61f2e08c1273b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41553 } } peers { permanent_uuid: "adcdc77475674133ac1b3086d879e85d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40117 } } peers { permanent_uuid: "15383ec9a2624d08974db8ee13127745" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33923 } }
I20260619 09:00:21.499895 22848 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15383ec9a2624d08974db8ee13127745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15383ec9a2624d08974db8ee13127745, State: Initialized, Role: FOLLOWER
I20260619 09:00:21.499949 22848 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15383ec9a2624d08974db8ee13127745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b42efdeaf244298b61f2e08c1273b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41553 } } peers { permanent_uuid: "adcdc77475674133ac1b3086d879e85d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40117 } } peers { permanent_uuid: "15383ec9a2624d08974db8ee13127745" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33923 } }
I20260619 09:00:21.500186 22855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15383ec9a2624d08974db8ee13127745 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b42efdeaf244298b61f2e08c1273b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41553 } } peers { permanent_uuid: "adcdc77475674133ac1b3086d879e85d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40117 } } peers { permanent_uuid: "15383ec9a2624d08974db8ee13127745" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33923 } } }
I20260619 09:00:21.500231 22848 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15383ec9a2624d08974db8ee13127745 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:00:21.500275 22855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15383ec9a2624d08974db8ee13127745 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 09:00:21.501077 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.493930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33923 (local address 127.0.0.1:55012)
0619 09:00:21.493962 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:21.493971 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 09:00:21.494004 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:00:21.494257 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:00:21.494258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:00:21.494396 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:00:21.494433 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:21.494435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.495299 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.495300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:21.496003 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:21.496006 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.496443 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.496444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:21.496502 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:21.496802 (+ 300us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:00:21.496806 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:00:21.497995 (+ 1189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:00:21.499846 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:21.499847 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:21.499847 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:21.500031 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:21.500484 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:21.500485 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:21.500486 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:21.500543 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:21.500781 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:00:21.500782 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:00:21.500824 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 09:00:21.500896 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 09:00:21.500988 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260619 09:00:21.501096 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.494004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55012 (local address 127.0.98.60:33923)
0619 09:00:21.494165 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:21.494166 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:21.494172 (+ 6us) server_negotiation.cc:407] Connection header received
0619 09:00:21.494185 (+ 13us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:21.494186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:21.494193 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:21.494244 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:21.494503 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.495219 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.496071 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.496374 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.496435 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:21.498080 (+ 1645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:21.498082 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:21.498082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:21.498088 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:21.499764 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:21.500349 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:21.500350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:21.500351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:21.500410 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:21.500593 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:21.500593 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:21.500594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:21.500726 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:21.500786 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:21.500918 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 09:00:21.500988 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":104,"threads_started":1}
I20260619 09:00:21.500780 22672 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "79b42efdeaf244298b61f2e08c1273b2"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52342
I20260619 09:00:21.502580 22698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79b42efdeaf244298b61f2e08c1273b2: Bootstrap starting.
I20260619 09:00:21.503309 22698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79b42efdeaf244298b61f2e08c1273b2: Neither blocks nor log segments found. Creating new log.
I20260619 09:00:21.503582 22698 log.cc:824] T 00000000000000000000000000000000 P 79b42efdeaf244298b61f2e08c1273b2: Log is configured to *not* fsync() on all Append() calls
W20260619 09:00:21.504381 22866 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 15383ec9a2624d08974db8ee13127745: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260619 09:00:21.504400 22698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79b42efdeaf244298b61f2e08c1273b2: No bootstrap required, opened a new log
W20260619 09:00:21.504436 22866 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 09:00:21.502271 22780 master_runner.cc:186] Error getting master registration for 127.0.98.62:41553: OK, instance_id { permanent_uuid: "79b42efdeaf244298b61f2e08c1273b2" instance_seqno: 1781859621154037 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 09:00:21.504918 22780 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 09:00:21.505736 22698 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79b42efdeaf244298b61f2e08c1273b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b42efdeaf244298b61f2e08c1273b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41553 } } peers { permanent_uuid: "adcdc77475674133ac1b3086d879e85d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40117 } } peers { permanent_uuid: "15383ec9a2624d08974db8ee13127745" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33923 } }
I20260619 09:00:21.505878 22698 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79b42efdeaf244298b61f2e08c1273b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79b42efdeaf244298b61f2e08c1273b2, State: Initialized, Role: FOLLOWER
I20260619 09:00:21.505963 22698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79b42efdeaf244298b61f2e08c1273b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b42efdeaf244298b61f2e08c1273b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41553 } } peers { permanent_uuid: "adcdc77475674133ac1b3086d879e85d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40117 } } peers { permanent_uuid: "15383ec9a2624d08974db8ee13127745" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33923 } }
I20260619 09:00:21.506210 22869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79b42efdeaf244298b61f2e08c1273b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b42efdeaf244298b61f2e08c1273b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41553 } } peers { permanent_uuid: "adcdc77475674133ac1b3086d879e85d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40117 } } peers { permanent_uuid: "15383ec9a2624d08974db8ee13127745" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33923 } } }
I20260619 09:00:21.506260 22698 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79b42efdeaf244298b61f2e08c1273b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:00:21.506307 22869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79b42efdeaf244298b61f2e08c1273b2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 09:00:21.506349 22672 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79b42efdeaf244298b61f2e08c1273b2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 09:00:21.506397 22672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79b42efdeaf244298b61f2e08c1273b2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:00:21.506932 22672 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79b42efdeaf244298b61f2e08c1273b2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b42efdeaf244298b61f2e08c1273b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41553 } } peers { permanent_uuid: "adcdc77475674133ac1b3086d879e85d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40117 } } peers { permanent_uuid: "15383ec9a2624d08974db8ee13127745" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33923 } }
I20260619 09:00:21.507097 22672 leader_election.cc:288] T 00000000000000000000000000000000 P 79b42efdeaf244298b61f2e08c1273b2 [CANDIDATE]: Term 1 election: Requested vote from peers adcdc77475674133ac1b3086d879e85d (127.0.98.61:40117), 15383ec9a2624d08974db8ee13127745 (127.0.98.60:33923)
I20260619 09:00:21.507579 22822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79b42efdeaf244298b61f2e08c1273b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "15383ec9a2624d08974db8ee13127745"
I20260619 09:00:21.507689 22822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15383ec9a2624d08974db8ee13127745 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:00:21.508580 22822 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 15383ec9a2624d08974db8ee13127745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79b42efdeaf244298b61f2e08c1273b2 in term 1.
I20260619 09:00:21.508932 22649 leader_election.cc:302] T 00000000000000000000000000000000 P 79b42efdeaf244298b61f2e08c1273b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15383ec9a2624d08974db8ee13127745, 79b42efdeaf244298b61f2e08c1273b2; no voters:
W20260619 09:00:21.508162 22880 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 79b42efdeaf244298b61f2e08c1273b2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 09:00:21.509127 22880 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 09:00:21.509534 22630 master_runner.cc:186] Error getting master registration for 127.0.98.61:40117: OK, instance_id { permanent_uuid: "adcdc77475674133ac1b3086d879e85d" instance_seqno: 1781859621311562 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 09:00:21.511698 22630 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 09:00:21.509449 22869 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79b42efdeaf244298b61f2e08c1273b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:00:21.511871 22869 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79b42efdeaf244298b61f2e08c1273b2 [term 1 LEADER]: Becoming Leader. State: Replica: 79b42efdeaf244298b61f2e08c1273b2, State: Running, Role: LEADER
I20260619 09:00:21.511965 22869 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79b42efdeaf244298b61f2e08c1273b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b42efdeaf244298b61f2e08c1273b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41553 } } peers { permanent_uuid: "adcdc77475674133ac1b3086d879e85d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40117 } } peers { permanent_uuid: "15383ec9a2624d08974db8ee13127745" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33923 } }
I20260619 09:00:21.512434 22886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79b42efdeaf244298b61f2e08c1273b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79b42efdeaf244298b61f2e08c1273b2. Latest consensus state: current_term: 1 leader_uuid: "79b42efdeaf244298b61f2e08c1273b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b42efdeaf244298b61f2e08c1273b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41553 } } peers { permanent_uuid: "adcdc77475674133ac1b3086d879e85d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40117 } } peers { permanent_uuid: "15383ec9a2624d08974db8ee13127745" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33923 } } }
I20260619 09:00:21.512523 22886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79b42efdeaf244298b61f2e08c1273b2 [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:21.523198 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.516347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52350 (local address 127.0.98.62:41553)
0619 09:00:21.516392 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:21.516394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:21.516494 (+ 100us) server_negotiation.cc:407] Connection header received
0619 09:00:21.516606 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:21.516607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:21.516617 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:21.516662 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:21.517372 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.517835 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.518802 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.518995 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.519046 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:21.520479 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:21.520480 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:21.520481 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:21.520487 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:21.522022 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:21.522378 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:21.522379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:21.522380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:21.522449 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:21.522631 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:21.522632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:21.522632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:21.522770 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:21.522863 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:21.523026 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 09:00:21.523104 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 09:00:21.525007 22888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:00:21.525223 22888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:00:21.528347 22822 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 15383ec9a2624d08974db8ee13127745 [term 1 FOLLOWER]: Refusing update from remote peer 79b42efdeaf244298b61f2e08c1273b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:00:21.528740 22886 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 79b42efdeaf244298b61f2e08c1273b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15383ec9a2624d08974db8ee13127745" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:00:21.530449 22855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15383ec9a2624d08974db8ee13127745 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79b42efdeaf244298b61f2e08c1273b2. Latest consensus state: current_term: 1 leader_uuid: "79b42efdeaf244298b61f2e08c1273b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b42efdeaf244298b61f2e08c1273b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41553 } } peers { permanent_uuid: "adcdc77475674133ac1b3086d879e85d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40117 } } peers { permanent_uuid: "15383ec9a2624d08974db8ee13127745" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33923 } } }
I20260619 09:00:21.530545 22855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15383ec9a2624d08974db8ee13127745 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 09:00:21.532424 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.516909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55018 (local address 127.0.98.60:33923)
0619 09:00:21.516956 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:21.516958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:21.524212 (+ 7254us) server_negotiation.cc:407] Connection header received
0619 09:00:21.524275 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:21.524276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:21.524288 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:21.524336 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:21.524959 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.525709 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.526711 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.527039 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.527101 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:21.529587 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:21.529588 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:21.529588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:21.529596 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:21.531153 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:21.531515 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:21.531516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:21.531516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:21.531574 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:21.531736 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:21.531737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:21.531737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:21.531885 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:21.532019 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:21.532250 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 09:00:21.532330 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 09:00:21.535946 22778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.516578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46440 (local address 127.0.98.61:40117)
0619 09:00:21.516621 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:21.516623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:21.523584 (+ 6961us) server_negotiation.cc:407] Connection header received
0619 09:00:21.523656 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:21.523657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:21.523668 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:21.523696 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:21.524317 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.525023 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.525919 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.526223 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.526282 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:21.533283 (+ 7001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:21.533285 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:21.533285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:21.533293 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:21.534741 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:21.535121 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:21.535122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:21.535123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:21.535195 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:21.535356 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:21.535357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:21.535357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:21.535504 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:21.535598 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:21.535763 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 09:00:21.535835 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 09:00:21.541253 22886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79b42efdeaf244298b61f2e08c1273b2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79b42efdeaf244298b61f2e08c1273b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b42efdeaf244298b61f2e08c1273b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41553 } } peers { permanent_uuid: "adcdc77475674133ac1b3086d879e85d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40117 } } peers { permanent_uuid: "15383ec9a2624d08974db8ee13127745" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33923 } } }
I20260619 09:00:21.541366 22886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79b42efdeaf244298b61f2e08c1273b2 [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:21.541734 22888 catalog_manager.cc:1347] Generated new cluster ID: 8a912de4b62c49d9bc2ee4783d93abe4
I20260619 09:00:21.541780 22888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:00:21.541967 22855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15383ec9a2624d08974db8ee13127745 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79b42efdeaf244298b61f2e08c1273b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b42efdeaf244298b61f2e08c1273b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41553 } } peers { permanent_uuid: "adcdc77475674133ac1b3086d879e85d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40117 } } peers { permanent_uuid: "15383ec9a2624d08974db8ee13127745" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33923 } } }
I20260619 09:00:21.542058 22855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15383ec9a2624d08974db8ee13127745 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 09:00:21.548360 22893 mvcc.cc:205] Tried to move back new op lower bound from 7298497009777012736 to 7298497009714606080. Current Snapshot: MvccSnapshot[applied={T|T < 7298497009777012736}]
I20260619 09:00:21.548833 22773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33923 } attempt: 3
Jun 19 09:00:21 dist-test-slave-h4g5 krb5kdc[22617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 09:00:21.556821 22776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.549086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33923 (local address 127.0.0.1:55026)
0619 09:00:21.549133 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:21.549143 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:00:21.549176 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:00:21.549310 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:00:21.549310 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:00:21.549481 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:00:21.549527 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:21.549529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.550377 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.550377 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:21.551174 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:21.551176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.551597 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.551597 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:21.551648 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:21.552120 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:00:21.552126 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:00:21.553936 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:00:21.555777 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:21.555779 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:21.555779 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:21.555981 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:21.556188 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:21.556189 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:21.556189 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:21.556244 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:21.556519 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:00:21.556519 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:00:21.556565 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 09:00:21.556624 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 09:00:21.556718 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 09:00:21.557093 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:21.549143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55026 (local address 127.0.98.60:33923)
0619 09:00:21.549182 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:21.549184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:21.549195 (+ 11us) server_negotiation.cc:407] Connection header received
0619 09:00:21.549218 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:21.549219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:21.549228 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:21.549293 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:21.549577 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.550311 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.551232 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:21.551534 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:21.551600 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:21.554000 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:21.554001 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:21.554002 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:21.554008 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:21.555688 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:21.556052 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:21.556054 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:21.556054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:21.556124 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:21.556304 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:21.556304 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:21.556305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:21.556428 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:21.556494 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:21.556932 (+ 438us) server_negotiation.cc:299] Negotiation successful
0619 09:00:21.557005 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 09:00:21.558614 22773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adcdc77475674133ac1b3086d879e85d: Bootstrap starting.
I20260619 09:00:21.559278 22773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adcdc77475674133ac1b3086d879e85d: Neither blocks nor log segments found. Creating new log.
I20260619 09:00:21.559548 22773 log.cc:824] T 00000000000000000000000000000000 P adcdc77475674133ac1b3086d879e85d: Log is configured to *not* fsync() on all Append() calls
I20260619 09:00:21.560554 22773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adcdc77475674133ac1b3086d879e85d: No bootstrap required, opened a new log
I20260619 09:00:21.562391 22773 raft_consensus.cc:348] T 00000000000000000000000000000000 P adcdc77475674133ac1b3086d879e85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b42efdeaf244298b61f2e08c1273b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41553 } } peers { permanent_uuid: "adcdc77475674133ac1b3086d879e85d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40117 } } peers { permanent_uuid: "15383ec9a2624d08974db8ee13127745" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33923 } }
I20260619 09:00:21.562521 22773 raft_consensus.cc:729] T 00000000000000000000000000000000 P adcdc77475674133ac1b3086d879e85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adcdc77475674133ac1b3086d879e85d, State: Initialized, Role: FOLLOWER
I20260619 09:00:21.562587 22773 consensus_queue.cc:260] T 00000000000000000000000000000000 P adcdc77475674133ac1b3086d879e85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b42efdeaf244298b61f2e08c1273b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41553 } } peers { permanent_uuid: "adcdc77475674133ac1b3086d879e85d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40117 } } peers { permanent_uuid: "15383ec9a2624d08974db8ee13127745" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33923 } }
I20260619 09:00:21.562878 22773 sys_catalog.cc:564] T 00000000000000000000000000000000 P adcdc77475674133ac1b3086d879e85d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:00:21.564049 22746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adcdc77475674133ac1b3086d879e85d [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:00:21.564831 22747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79b42efdeaf244298b61f2e08c1273b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "adcdc77475674133ac1b3086d879e85d"
I20260619 09:00:21.565562 22746 raft_consensus.cc:1261] T 00000000000000000000000000000000 P adcdc77475674133ac1b3086d879e85d [term 1 FOLLOWER]: Refusing update from remote peer 79b42efdeaf244298b61f2e08c1273b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:00:21.565898 22869 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 79b42efdeaf244298b61f2e08c1273b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adcdc77475674133ac1b3086d879e85d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:00:21.566844 22747 raft_consensus.cc:2454] T 00000000000000000000000000000000 P adcdc77475674133ac1b3086d879e85d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79b42efdeaf244298b61f2e08c1273b2 in term 1.
I20260619 09:00:21.570299 22898 sys_catalog.cc:455] T 00000000000000000000000000000000 P adcdc77475674133ac1b3086d879e85d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79b42efdeaf244298b61f2e08c1273b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b42efdeaf244298b61f2e08c1273b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41553 } } peers { permanent_uuid: "adcdc77475674133ac1b3086d879e85d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40117 } } peers { permanent_uuid: "15383ec9a2624d08974db8ee13127745" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33923 } } }
I20260619 09:00:21.570978 22898 sys_catalog.cc:458] T 00000000000000000000000000000000 P adcdc77475674133ac1b3086d879e85d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 09:00:21.571064 22898 sys_catalog.cc:455] T 00000000000000000000000000000000 P adcdc77475674133ac1b3086d879e85d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79b42efdeaf244298b61f2e08c1273b2. Latest consensus state: current_term: 1 leader_uuid: "79b42efdeaf244298b61f2e08c1273b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b42efdeaf244298b61f2e08c1273b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41553 } } peers { permanent_uuid: "adcdc77475674133ac1b3086d879e85d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40117 } } peers { permanent_uuid: "15383ec9a2624d08974db8ee13127745" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33923 } } }
I20260619 09:00:21.571156 22898 sys_catalog.cc:458] T 00000000000000000000000000000000 P adcdc77475674133ac1b3086d879e85d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 09:00:21.571213 22898 sys_catalog.cc:455] T 00000000000000000000000000000000 P adcdc77475674133ac1b3086d879e85d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79b42efdeaf244298b61f2e08c1273b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b42efdeaf244298b61f2e08c1273b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41553 } } peers { permanent_uuid: "adcdc77475674133ac1b3086d879e85d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40117 } } peers { permanent_uuid: "15383ec9a2624d08974db8ee13127745" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33923 } } }
I20260619 09:00:21.571293 22898 sys_catalog.cc:458] T 00000000000000000000000000000000 P adcdc77475674133ac1b3086d879e85d [sys.catalog]: This master's current role is: FOLLOWER
W20260619 09:00:21.570923 22909 catalog_manager.cc:1558] T 00000000000000000000000000000000 P adcdc77475674133ac1b3086d879e85d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 09:00:21.571471 22909 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 09:00:21.574870 22911 mvcc.cc:205] Tried to move back new op lower bound from 7298497009777012736 to 7298497009714606080. Current Snapshot: MvccSnapshot[applied={T|T < 7298497009777012736}]
I20260619 09:00:21.581084 22869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79b42efdeaf244298b61f2e08c1273b2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79b42efdeaf244298b61f2e08c1273b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b42efdeaf244298b61f2e08c1273b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41553 } } peers { permanent_uuid: "adcdc77475674133ac1b3086d879e85d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40117 } } peers { permanent_uuid: "15383ec9a2624d08974db8ee13127745" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33923 } } }
I20260619 09:00:21.581215 22869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79b42efdeaf244298b61f2e08c1273b2 [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:21.582593 22888 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:00:21.583288 22888 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:00:21.596166 22888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79b42efdeaf244298b61f2e08c1273b2: Generated new TSK 0
I20260619 09:00:21.596370 22888 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 09:00:21.620618 22778 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:00:21.619042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46448 (local address 127.0.98.61:40117)
0619 09:00:21.619087 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:21.619089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:21.619960 (+ 871us) server_negotiation.cc:407] Connection header received
0619 09:00:21.620062 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:21.620064 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:21.620092 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:21.620122 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:21.620544 (+ 422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46448: BlockingRecv error: recv got EOF from 127.0.0.1:46448 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260619 09:00:21.621589 22701 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:00:21.619722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52356 (local address 127.0.98.62:41553)
0619 09:00:21.619768 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:21.619770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:21.620958 (+ 1188us) server_negotiation.cc:407] Connection header received
0619 09:00:21.621090 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:21.621091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:21.621102 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:21.621130 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:21.621530 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52356: BlockingRecv error: recv got EOF from 127.0.0.1:52356 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260619 09:00:21.622054 22853 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:00:21.619398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55036 (local address 127.0.98.60:33923)
0619 09:00:21.619439 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:21.619441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:21.620888 (+ 1447us) server_negotiation.cc:407] Connection header received
0619 09:00:21.621188 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:21.621189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:21.621199 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:21.621225 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:21.621997 (+ 772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55036: BlockingRecv error: recv got EOF from 127.0.0.1:55036 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260619 09:00:21.623934 22778 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:00:21.623317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46464 (local address 127.0.98.61:40117)
0619 09:00:21.623355 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:21.623356 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:21.623478 (+ 122us) server_negotiation.cc:407] Connection header received
0619 09:00:21.623576 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:21.623577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:21.623594 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:21.623621 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:21.623885 (+ 264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46464: BlockingRecv error: recv got EOF from 127.0.0.1:46464 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260619 09:00:21.626803 22778 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:00:21.626156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46474 (local address 127.0.98.61:40117)
0619 09:00:21.626192 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:21.626194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:21.626340 (+ 146us) server_negotiation.cc:407] Connection header received
0619 09:00:21.626441 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:21.626442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:21.626452 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:21.626479 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:21.626759 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46474: BlockingRecv error: recv got EOF from 127.0.0.1:46474 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260619 09:00:21.630589 22778 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:00:21.629876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46486 (local address 127.0.98.61:40117)
0619 09:00:21.629908 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:21.629909 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:21.630128 (+ 219us) server_negotiation.cc:407] Connection header received
0619 09:00:21.630214 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:21.630215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:21.630225 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:21.630252 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:21.630544 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46486: BlockingRecv error: recv got EOF from 127.0.0.1:46486 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260619 09:00:21.636523 22778 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:00:21.635684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46496 (local address 127.0.98.61:40117)
0619 09:00:21.635719 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:21.635720 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:21.635965 (+ 245us) server_negotiation.cc:407] Connection header received
0619 09:00:21.636057 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:21.636058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:21.636068 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:21.636137 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:21.636469 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46496: BlockingRecv error: recv got EOF from 127.0.0.1:46496 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260619 09:00:21.646468 22778 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:00:21.645645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46508 (local address 127.0.98.61:40117)
0619 09:00:21.645683 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:21.645685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:21.645933 (+ 248us) server_negotiation.cc:407] Connection header received
0619 09:00:21.646027 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:21.646028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:21.646039 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:21.646067 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:21.646401 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46508: BlockingRecv error: recv got EOF from 127.0.0.1:46508 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260619 09:00:21.664780 22778 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:00:21.663771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46520 (local address 127.0.98.61:40117)
0619 09:00:21.663814 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:21.663816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:21.664157 (+ 341us) server_negotiation.cc:407] Connection header received
0619 09:00:21.664267 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:21.664268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:21.664280 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:21.664309 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:21.664718 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46520: BlockingRecv error: recv got EOF from 127.0.0.1:46520 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260619 09:00:21.699122 22778 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:00:21.698126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47740 (local address 127.0.98.61:40117)
0619 09:00:21.698178 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:21.698180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:21.698508 (+ 328us) server_negotiation.cc:407] Connection header received
0619 09:00:21.698617 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:21.698618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:21.698631 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:21.698661 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:21.699060 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47740: BlockingRecv error: recv got EOF from 127.0.0.1:47740 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260619 09:00:21.765213 22778 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:00:21.764532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47742 (local address 127.0.98.61:40117)
0619 09:00:21.764579 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:21.764581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:21.764721 (+ 140us) server_negotiation.cc:407] Connection header received
0619 09:00:21.764784 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:21.764785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:21.764796 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:21.764845 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:21.765151 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47742: BlockingRecv error: recv got EOF from 127.0.0.1:47742 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260619 09:00:21.895045 22778 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:00:21.894307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47756 (local address 127.0.98.61:40117)
0619 09:00:21.894375 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:21.894377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:21.894509 (+ 132us) server_negotiation.cc:407] Connection header received
0619 09:00:21.894542 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:21.894542 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:21.894554 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:21.894613 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:21.894995 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47756: BlockingRecv error: recv got EOF from 127.0.0.1:47756 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 09:00:22.152909 22778 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:00:22.152269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47772 (local address 127.0.98.61:40117)
0619 09:00:22.152335 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:22.152337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:22.152403 (+ 66us) server_negotiation.cc:407] Connection header received
0619 09:00:22.152454 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:22.152455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:22.152465 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:22.152516 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:22.152848 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47772: BlockingRecv error: recv got EOF from 127.0.0.1:47772 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 09:00:22.504882 22866 catalog_manager.cc:1259] Loaded cluster ID: 8a912de4b62c49d9bc2ee4783d93abe4
I20260619 09:00:22.504966 22866 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 15383ec9a2624d08974db8ee13127745: loading cluster ID for follower catalog manager: success
I20260619 09:00:22.505790 22866 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 15383ec9a2624d08974db8ee13127745: acquiring CA information for follower catalog manager: success
I20260619 09:00:22.506072 22866 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 15383ec9a2624d08974db8ee13127745: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 09:00:22.571934 22909 catalog_manager.cc:1259] Loaded cluster ID: 8a912de4b62c49d9bc2ee4783d93abe4
I20260619 09:00:22.572014 22909 catalog_manager.cc:1552] T 00000000000000000000000000000000 P adcdc77475674133ac1b3086d879e85d: loading cluster ID for follower catalog manager: success
I20260619 09:00:22.572853 22909 catalog_manager.cc:1574] T 00000000000000000000000000000000 P adcdc77475674133ac1b3086d879e85d: acquiring CA information for follower catalog manager: success
I20260619 09:00:22.573159 22909 catalog_manager.cc:1602] T 00000000000000000000000000000000 P adcdc77475674133ac1b3086d879e85d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 09:00:22.668591 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:22.666114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47774 (local address 127.0.98.61:40117)
0619 09:00:22.666233 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:22.666235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:22.666279 (+ 44us) server_negotiation.cc:407] Connection header received
0619 09:00:22.666334 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:22.666335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:22.666383 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:22.666455 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:00:22.666574 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:22.667058 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:22.667807 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:22.668048 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:22.668181 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:22.668310 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:00:22.668419 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:00:22.668471 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:22.668498 (+ 27us) server_negotiation.cc:299] Negotiation successful
0619 09:00:22.668517 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":71,"threads_started":1}
I20260619 09:00:22.672611 22987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:22.670321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39762 (local address 127.0.98.60:33923)
0619 09:00:22.670447 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:22.670448 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:22.670456 (+ 8us) server_negotiation.cc:407] Connection header received
0619 09:00:22.670493 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:22.670494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:22.670527 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:22.670603 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:00:22.670693 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:22.671187 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:22.671923 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:22.672179 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:22.672236 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:22.672349 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:00:22.672448 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:00:22.672515 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:22.672528 (+ 13us) server_negotiation.cc:299] Negotiation successful
0619 09:00:22.672546 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":75,"threads_started":1}
I20260619 09:00:22.677053 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:22.674244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47780 (local address 127.0.98.61:40117)
0619 09:00:22.674295 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:22.674297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:22.674384 (+ 87us) server_negotiation.cc:407] Connection header received
0619 09:00:22.674474 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:22.674475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:22.674486 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:22.674511 (+ 25us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:00:22.674700 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:22.675204 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:22.676357 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:22.676576 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:22.676631 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:22.676792 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:00:22.676825 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:00:22.676852 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:22.676952 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 09:00:22.676978 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 09:00:22.677416 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:22.674307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33012 (local address 127.0.98.62:41553)
0619 09:00:22.674404 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:22.674406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:22.674492 (+ 86us) server_negotiation.cc:407] Connection header received
0619 09:00:22.674537 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:22.674538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:22.674585 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:22.674700 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:00:22.674807 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:22.675295 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:22.676340 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:22.676656 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:22.676721 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:22.676909 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:00:22.677029 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:00:22.677294 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:22.677311 (+ 17us) server_negotiation.cc:299] Negotiation successful
0619 09:00:22.677336 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":46,"threads_started":1}
I20260619 09:00:22.677493 22987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:22.674443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39772 (local address 127.0.98.60:33923)
0619 09:00:22.674479 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:22.674481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:22.674596 (+ 115us) server_negotiation.cc:407] Connection header received
0619 09:00:22.674642 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:22.674643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:22.674651 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:22.674698 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:00:22.675399 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:22.675885 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:22.676805 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:22.677053 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:22.677113 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:22.677250 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:00:22.677283 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:00:22.677310 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:22.677407 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 09:00:22.677429 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 09:00:22.678848 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 22630
I20260619 09:00:22.683269 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 22705
I20260619 09:00:22.687165 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 22780
2026-06-19T09:00:22Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2728 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:00:22 dist-test-slave-h4g5 krb5kdc[22998](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:00:22 dist-test-slave-h4g5 krb5kdc[22998](info): set up 2 sockets
Jun 19 09:00:22 dist-test-slave-h4g5 krb5kdc[22998](info): commencing operation
krb5kdc: starting...
W20260619 09:00:23.632028 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/krb5kdc/test-user.keytab.
Jun 19 09:00:23 dist-test-slave-h4g5 krb5kdc[22998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:00:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:00:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 09:00:23.751036 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40491
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40491,127.0.98.61:33421,127.0.98.60:35569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:00:23.830680 23011 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 09:00:23.830904 23011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:00:23.830919 23011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:00:23.830941 23011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:00:23.832130 23011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:00:23.832172 23011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:00:23.832185 23011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:00:23.832196 23011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:00:23.832207 23011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:00:23.833720 23011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40491,127.0.98.61:33421,127.0.98.60:35569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:00:23.833961 23011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:00:23.834179 23011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:00:23.839116 23016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:23.839141 23017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:00:23.839178 23011 server_base.cc:1034] running on GCE node
W20260619 09:00:23.839152 23019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:00:23.839596 23011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:00:23.839902 23011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:00:23.841050 23011 hybrid_clock.cc:648] HybridClock initialized: now 1781859623841023 us; error 48 us; skew 500 ppm
Jun 19 09:00:23 dist-test-slave-h4g5 krb5kdc[22998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:00:23.848332 23011 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:00:23.848796 23011 webserver.cc:460] Webserver started at http://127.0.98.62:34633/ using document root <none> and password file <none>
I20260619 09:00:23.848999 23011 fs_manager.cc:359] Metadata directory not provided
I20260619 09:00:23.849052 23011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:00:23.849174 23011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:00:23.850492 23011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-0/data/instance:
uuid: "9a709cbcfe8b416c9a752dbf3d4005ca"
format_stamp: "Formatted at 2026-06-19 09:00:23 on dist-test-slave-h4g5"
server_key: "70d13281d3d867fce554199978c8612e"
server_key_iv: "6ff5c0073f3f3f394ecff39e758490b2"
server_key_version: "encryptionkey@0"
I20260619 09:00:23.850833 23011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-0/wal/instance:
uuid: "9a709cbcfe8b416c9a752dbf3d4005ca"
format_stamp: "Formatted at 2026-06-19 09:00:23 on dist-test-slave-h4g5"
server_key: "70d13281d3d867fce554199978c8612e"
server_key_iv: "6ff5c0073f3f3f394ecff39e758490b2"
server_key_version: "encryptionkey@0"
I20260619 09:00:23.852138 23011 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:00:23.852845 23026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:00:23.852982 23011 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:00:23.853078 23011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-0/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-0/wal
uuid: "9a709cbcfe8b416c9a752dbf3d4005ca"
format_stamp: "Formatted at 2026-06-19 09:00:23 on dist-test-slave-h4g5"
server_key: "70d13281d3d867fce554199978c8612e"
server_key_iv: "6ff5c0073f3f3f394ecff39e758490b2"
server_key_version: "encryptionkey@0"
I20260619 09:00:23.853142 23011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:00:23.869197 23011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:00:23.871205 23011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:00:23.871340 23011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:00:23.875276 23011 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40491
I20260619 09:00:23.875322 23078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40491 every 8 connection(s)
I20260619 09:00:23.875655 23011 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:00:23.876652 23079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:00:23.878167 23079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 09:00:23 dist-test-slave-h4g5 krb5kdc[22998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:00:23.885879 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 23011
I20260619 09:00:23.885980 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-0/wal/instance
I20260619 09:00:23.886148 392 external_mini_cluster.cc:1428] Setting key 5afb18abf9f24dd6cf7e33b352e24b04
I20260619 09:00:23.887977 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:23.878809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40491 (local address 127.0.0.1:40648)
0619 09:00:23.879033 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:23.879073 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 09:00:23.879618 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:00:23.879932 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:00:23.879935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:00:23.880261 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:00:23.880472 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:23.880477 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:23.881501 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:23.881502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:23.882196 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:23.882198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:23.882703 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:23.882705 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:23.882763 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:23.883220 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:00:23.883228 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:00:23.884689 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:00:23.886349 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:23.886356 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:23.886359 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:23.886631 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:23.886893 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:23.886893 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:23.886894 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:23.886993 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:23.887481 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:00:23.887488 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:00:23.887570 (+ 82us) client_negotiation.cc:769] Sending connection context
0619 09:00:23.887647 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 09:00:23.887779 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":93,"threads_started":1}
I20260619 09:00:23.888008 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:23.878844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40648 (local address 127.0.98.62:40491)
0619 09:00:23.879054 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:23.879061 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:23.879161 (+ 100us) server_negotiation.cc:407] Connection header received
0619 09:00:23.879773 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:23.879775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:23.879793 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:23.879874 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:23.880570 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:23.881369 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:23.882275 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:23.882607 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:23.882685 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:23.884769 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:23.884776 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:23.884776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:23.884782 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:23.886276 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:23.886731 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:23.886735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:23.886735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:23.886827 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:23.887081 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:23.887082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:23.887082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:23.887395 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:23.887501 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:23.887692 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 09:00:23.887866 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":184,"thread_start_us":75,"threads_started":1}
I20260619 09:00:23.888811 23079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33421 } has no permanent_uuid. Determining permanent_uuid...
I20260619 09:00:23.889111 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:23.889010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33421 (local address 127.0.0.1:50950)
0619 09:00:23.889051 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:23.889092 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260619 09:00:23.889235 23029 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33421: connect: Connection refused (error 111)
I20260619 09:00:23.889662 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:23.889581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33421 (local address 127.0.0.1:50952)
0619 09:00:23.889623 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:23.889646 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 09:00:23.889777 23079 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33421: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33421: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 09:00:23.910914 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33421
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40491,127.0.98.61:33421,127.0.98.60:35569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 09:00:23.939021 23079 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33421 } attempt: 1
I20260619 09:00:23.939616 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:23.939445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33421 (local address 127.0.0.1:50960)
0619 09:00:23.939518 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:23.939575 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 09:00:23.940025 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:23.939924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33421 (local address 127.0.0.1:50968)
0619 09:00:23.939966 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:23.939999 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 09:00:23.940135 23079 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33421: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33421: connect: Connection refused (error 111)
W20260619 09:00:23.992838 23086 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 09:00:23.993058 23086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:00:23.993074 23086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:00:23.993094 23086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:00:23.994261 23086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:00:23.994308 23086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:00:23.994321 23086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:00:23.994333 23086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:00:23.994343 23086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:00:23.995888 23086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40491,127.0.98.61:33421,127.0.98.60:35569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:00:23.996157 23086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:00:23.996376 23086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 09:00:24.001098 23086 server_base.cc:1034] running on GCE node
W20260619 09:00:24.001062 23091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:24.001063 23092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:24.001063 23094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:00:24.001513 23086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:00:24.001770 23086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:00:24.002992 23086 hybrid_clock.cc:648] HybridClock initialized: now 1781859624002964 us; error 54 us; skew 500 ppm
Jun 19 09:00:24 dist-test-slave-h4g5 krb5kdc[22998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:00:24.005568 23086 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 09:00:24.006064 23086 webserver.cc:460] Webserver started at http://127.0.98.61:42537/ using document root <none> and password file <none>
I20260619 09:00:24.006281 23086 fs_manager.cc:359] Metadata directory not provided
I20260619 09:00:24.006330 23086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:00:24.006450 23086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:00:24.007503 23086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-1/data/instance:
uuid: "f05e92e62bec49ecbfb26442e3f3e8f6"
format_stamp: "Formatted at 2026-06-19 09:00:24 on dist-test-slave-h4g5"
server_key: "5e7d8d14f9b29489a24b2577defca7a9"
server_key_iv: "ea225509e2bd9e86a714bc65a711c7dc"
server_key_version: "encryptionkey@0"
I20260619 09:00:24.007843 23086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-1/wal/instance:
uuid: "f05e92e62bec49ecbfb26442e3f3e8f6"
format_stamp: "Formatted at 2026-06-19 09:00:24 on dist-test-slave-h4g5"
server_key: "5e7d8d14f9b29489a24b2577defca7a9"
server_key_iv: "ea225509e2bd9e86a714bc65a711c7dc"
server_key_version: "encryptionkey@0"
I20260619 09:00:24.008270 23079 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33421 } attempt: 2
I20260619 09:00:24.008663 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:24.008521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33421 (local address 127.0.0.1:50978)
0619 09:00:24.008588 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:24.008628 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 09:00:24.008972 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:24.008879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33421 (local address 127.0.0.1:50988)
0619 09:00:24.008922 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:24.008948 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260619 09:00:24.009065 23079 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33421: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33421: connect: Connection refused (error 111)
I20260619 09:00:24.009436 23086 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:00:24.010152 23101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:00:24.010278 23086 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:00:24.010357 23086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-1/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-1/wal
uuid: "f05e92e62bec49ecbfb26442e3f3e8f6"
format_stamp: "Formatted at 2026-06-19 09:00:24 on dist-test-slave-h4g5"
server_key: "5e7d8d14f9b29489a24b2577defca7a9"
server_key_iv: "ea225509e2bd9e86a714bc65a711c7dc"
server_key_version: "encryptionkey@0"
I20260619 09:00:24.010428 23086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:00:24.023700 23086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:00:24.025966 23086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:00:24.026104 23086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:00:24.030050 23086 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33421
I20260619 09:00:24.030087 23153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33421 every 8 connection(s)
I20260619 09:00:24.030426 23086 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-1/data/info.pb
I20260619 09:00:24.031385 23154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:00:24.032991 23154 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } has no permanent_uuid. Determining permanent_uuid...
I20260619 09:00:24.035861 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 23086
I20260619 09:00:24.035981 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-1/wal/instance
I20260619 09:00:24.036209 392 external_mini_cluster.cc:1428] Setting key 7457a73ed398bea388610f5df4d68d83
Jun 19 09:00:24 dist-test-slave-h4g5 krb5kdc[22998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:00:24.042960 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:24.033742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40658 (local address 127.0.98.62:40491)
0619 09:00:24.033822 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:24.033824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:24.033977 (+ 153us) server_negotiation.cc:407] Connection header received
0619 09:00:24.034506 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:24.034507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:24.034523 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:24.034660 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:24.035407 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.036009 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.037489 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.037804 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.037881 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:24.040251 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:24.040253 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:24.040253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:24.040261 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:24.041641 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:24.042044 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:24.042045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:24.042046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:24.042100 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:24.042335 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:24.042336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:24.042336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:24.042451 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:24.042506 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:24.042775 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 09:00:24.042861 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 09:00:24.043139 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:24.033665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40491 (local address 127.0.0.1:40658)
0619 09:00:24.033839 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:24.033880 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 09:00:24.034380 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:00:24.034734 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:00:24.034737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:00:24.035023 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:00:24.035269 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:24.035276 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.036136 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.036138 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:24.037345 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:24.037349 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.037926 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.037928 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:24.038052 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:24.038568 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:00:24.038578 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:00:24.040159 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:00:24.041729 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:24.041738 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:24.041741 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:24.041973 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:24.042156 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:24.042157 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:24.042158 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:24.042280 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:24.042505 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:00:24.042512 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:00:24.042655 (+ 143us) client_negotiation.cc:769] Sending connection context
0619 09:00:24.042747 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 09:00:24.042914 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":71,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260619 09:00:24.043716 23154 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33421 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 19 09:00:24 dist-test-slave-h4g5 krb5kdc[22998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 09:00:24.051026 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:24.043972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33421 (local address 127.0.0.1:51000)
0619 09:00:24.044013 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:24.044024 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:00:24.044061 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:00:24.044380 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:00:24.044381 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:00:24.044501 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:00:24.044554 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:24.044556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.045479 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.045479 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:24.046200 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:24.046202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.046811 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.046811 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:24.046884 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:24.047296 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:00:24.047301 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:00:24.048535 (+ 1234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:00:24.050093 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:24.050093 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:24.050094 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:24.050241 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:24.050421 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:24.050421 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:24.050422 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:24.050474 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:24.050716 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:00:24.050717 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:00:24.050755 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 09:00:24.050813 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 09:00:24.050913 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 09:00:24.051084 23158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:24.044034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51000 (local address 127.0.98.61:33421)
0619 09:00:24.044199 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:24.044203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:24.044228 (+ 25us) server_negotiation.cc:407] Connection header received
0619 09:00:24.044249 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:24.044252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:24.044265 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:24.044346 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:24.044631 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.045381 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.046268 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.046743 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.046805 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:24.048606 (+ 1801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:24.048615 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:24.048616 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:24.048621 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:24.050019 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:24.050305 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:24.050308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:24.050308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:24.050362 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:24.050523 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:24.050524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:24.050524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:24.050658 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:24.050731 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:24.050843 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 09:00:24.050968 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":136,"thread_start_us":102,"threads_started":1}
I20260619 09:00:24.051595 23154 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } has no permanent_uuid. Determining permanent_uuid...
I20260619 09:00:24.051940 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:24.051828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35569 (local address 127.0.0.1:56112)
0619 09:00:24.051868 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:24.051911 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260619 09:00:24.052052 23105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35569: connect: Connection refused (error 111)
I20260619 09:00:24.052455 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:24.052373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35569 (local address 127.0.0.1:56120)
0619 09:00:24.052408 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:24.052431 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260619 09:00:24.052554 23154 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35569: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35569: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 09:00:24.060351 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5ocqKJ/build/release/bin/kudu
/tmp/dist-test-task5ocqKJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35569
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40491,127.0.98.61:33421,127.0.98.60:35569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 09:00:24.100261 23079 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33421 } attempt: 3
I20260619 09:00:24.101799 23154 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } attempt: 1
I20260619 09:00:24.102273 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:24.102076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35569 (local address 127.0.0.1:56126)
0619 09:00:24.102136 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:24.102233 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 09:00:24.102674 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:24.102570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35569 (local address 127.0.0.1:56140)
0619 09:00:24.102618 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:24.102649 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260619 09:00:24.102772 23154 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35569: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35569: connect: Connection refused (error 111)
Jun 19 09:00:24 dist-test-slave-h4g5 krb5kdc[22998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 09:00:24.108361 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:24.100692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33421 (local address 127.0.0.1:51008)
0619 09:00:24.100781 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:24.100796 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 09:00:24.100871 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:00:24.101041 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:00:24.101042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:00:24.101255 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:00:24.101338 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:24.101341 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.102076 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.102078 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:24.103199 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:24.103203 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.103760 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.103761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:24.103841 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:24.104308 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:00:24.104313 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:00:24.105653 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:00:24.107337 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:24.107338 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:24.107338 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:24.107510 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:24.107701 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:24.107701 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:24.107702 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:24.107775 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:24.108040 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:00:24.108041 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:00:24.108118 (+ 77us) client_negotiation.cc:769] Sending connection context
0619 09:00:24.108183 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 09:00:24.108282 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 09:00:24.108393 23158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:24.100801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51008 (local address 127.0.98.61:33421)
0619 09:00:24.100857 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:24.100859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:24.100870 (+ 11us) server_negotiation.cc:407] Connection header received
0619 09:00:24.100956 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:24.100956 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:24.100970 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:24.101028 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:24.101430 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.101947 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.103290 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.103624 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.103722 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:24.105742 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:24.105744 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:24.105744 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:24.105752 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:24.107244 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:24.107577 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:24.107579 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:24.107579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:24.107640 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:24.107828 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:24.107829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:24.107829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:24.107962 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:24.108024 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:24.108223 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 09:00:24.108306 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 09:00:24.108889 23079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } has no permanent_uuid. Determining permanent_uuid...
I20260619 09:00:24.109203 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:24.109108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35569 (local address 127.0.0.1:56156)
0619 09:00:24.109150 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:24.109177 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 09:00:24.109553 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:24.109438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35569 (local address 127.0.0.1:56160)
0619 09:00:24.109498 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:24.109530 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 09:00:24.109647 23079 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35569: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35569: connect: Connection refused (error 111)
I20260619 09:00:24.140797 23079 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } attempt: 1
I20260619 09:00:24.141276 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:24.141140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35569 (local address 127.0.0.1:56176)
0619 09:00:24.141191 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:24.141224 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 09:00:24.141630 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:24.141549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35569 (local address 127.0.0.1:56186)
0619 09:00:24.141588 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:24.141614 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 09:00:24.141714 23079 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35569: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35569: connect: Connection refused (error 111)
W20260619 09:00:24.142375 23161 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 09:00:24.142580 23161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:00:24.142604 23161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:00:24.142637 23161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:00:24.143841 23161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:00:24.143891 23161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:00:24.143910 23161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:00:24.143926 23161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:00:24.143941 23161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:00:24.145550 23161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40491,127.0.98.61:33421,127.0.98.60:35569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:00:24.145805 23161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:00:24.146018 23161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:00:24.150862 23167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:00:24.151000 23161 server_base.cc:1034] running on GCE node
W20260619 09:00:24.150914 23166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:24.150862 23169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:00:24.151413 23161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:00:24.151688 23161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:00:24.152850 23161 hybrid_clock.cc:648] HybridClock initialized: now 1781859624152834 us; error 32 us; skew 500 ppm
Jun 19 09:00:24 dist-test-slave-h4g5 krb5kdc[22998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:00:24.155293 23161 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 09:00:24.155699 23161 webserver.cc:460] Webserver started at http://127.0.98.60:40221/ using document root <none> and password file <none>
I20260619 09:00:24.155957 23161 fs_manager.cc:359] Metadata directory not provided
I20260619 09:00:24.156011 23161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:00:24.156158 23161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:00:24.157166 23161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-2/data/instance:
uuid: "2622848d3442475bb10cd797468d22d3"
format_stamp: "Formatted at 2026-06-19 09:00:24 on dist-test-slave-h4g5"
server_key: "87c309311e7e48e5a838dfe981da938a"
server_key_iv: "35aa41e00e12bb5f72675d721d373b4b"
server_key_version: "encryptionkey@0"
I20260619 09:00:24.157548 23161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-2/wal/instance:
uuid: "2622848d3442475bb10cd797468d22d3"
format_stamp: "Formatted at 2026-06-19 09:00:24 on dist-test-slave-h4g5"
server_key: "87c309311e7e48e5a838dfe981da938a"
server_key_iv: "35aa41e00e12bb5f72675d721d373b4b"
server_key_version: "encryptionkey@0"
I20260619 09:00:24.159111 23161 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:00:24.159852 23176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:00:24.160017 23161 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:00:24.160130 23161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-2/data,/tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-2/wal
uuid: "2622848d3442475bb10cd797468d22d3"
format_stamp: "Formatted at 2026-06-19 09:00:24 on dist-test-slave-h4g5"
server_key: "87c309311e7e48e5a838dfe981da938a"
server_key_iv: "35aa41e00e12bb5f72675d721d373b4b"
server_key_version: "encryptionkey@0"
I20260619 09:00:24.160269 23161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:00:24.167156 23161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:00:24.169160 23161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:00:24.169282 23161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:00:24.170923 23154 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } attempt: 2
I20260619 09:00:24.171396 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:24.171190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35569 (local address 127.0.0.1:56194)
0619 09:00:24.171306 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:24.171354 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 09:00:24.171801 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:24.171672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35569 (local address 127.0.0.1:56206)
0619 09:00:24.171722 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:24.171753 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 09:00:24.171893 23154 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35569: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35569: connect: Connection refused (error 111)
I20260619 09:00:24.173110 23161 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35569
I20260619 09:00:24.173141 23228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35569 every 8 connection(s)
I20260619 09:00:24.173453 23161 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-2/data/info.pb
I20260619 09:00:24.174396 23229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:00:24.174916 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5ocqKJ/build/release/bin/kudu as pid 23161
I20260619 09:00:24.175002 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5ocqKJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859361540516-392-0/minicluster-data/master-2/wal/instance
I20260619 09:00:24.175154 392 external_mini_cluster.cc:1428] Setting key ade9231b345462cf8212f5c3abf0b9a0
I20260619 09:00:24.176240 23229 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 09:00:24 dist-test-slave-h4g5 krb5kdc[22998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 09:00:24 dist-test-slave-h4g5 krb5kdc[22998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 09:00:24.184605 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:24.177053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40666 (local address 127.0.98.62:40491)
0619 09:00:24.177093 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:24.177095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:24.177244 (+ 149us) server_negotiation.cc:407] Connection header received
0619 09:00:24.177308 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:24.177309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:24.177320 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:24.177364 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:24.177953 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.178416 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.179299 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.179508 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.179557 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:24.181800 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:24.181802 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:24.181802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:24.181810 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:24.183247 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:24.183651 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:24.183651 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:24.183652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:24.183700 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:24.183946 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:24.183947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:24.183948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:24.184107 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:24.184159 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:24.184449 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 09:00:24.184514 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 09:00:24.184841 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:24.177006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40491 (local address 127.0.0.1:40666)
0619 09:00:24.177150 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:24.177184 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 09:00:24.177247 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:00:24.177389 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:00:24.177392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:00:24.177750 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:00:24.177858 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:24.177864 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.178485 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.178486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:24.179179 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:24.179181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.179591 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.179592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:24.179691 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:24.180310 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:00:24.180322 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:00:24.181706 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:00:24.183332 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:24.183337 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:24.183340 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:24.183580 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:24.183751 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:24.183752 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:24.183752 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:24.183878 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:24.184168 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:00:24.184173 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:00:24.184318 (+ 145us) client_negotiation.cc:769] Sending connection context
0619 09:00:24.184427 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 09:00:24.184586 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":93,"threads_started":1}
I20260619 09:00:24.185262 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:24.175649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56220 (local address 127.0.98.60:35569)
0619 09:00:24.175842 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:24.175853 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:24.175885 (+ 32us) server_negotiation.cc:407] Connection header received
0619 09:00:24.176422 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:24.176431 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:24.176497 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:24.176643 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:24.177417 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.178488 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.179270 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.179585 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.179691 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:24.182225 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:24.182233 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:24.182233 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:24.182239 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:24.183754 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:24.184069 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:24.184072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:24.184072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:24.184151 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:24.184351 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:24.184352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:24.184353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:24.184500 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:24.184630 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:24.184925 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 09:00:24.185002 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":97,"threads_started":1}
I20260619 09:00:24.185369 23229 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33421 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 09:00:24 dist-test-slave-h4g5 krb5kdc[22998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 09:00:24 dist-test-slave-h4g5 krb5kdc[22998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:00:24.191565 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:24.185603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33421 (local address 127.0.0.1:51024)
0619 09:00:24.185646 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:24.185655 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 09:00:24.185697 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:00:24.185823 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:00:24.185824 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:00:24.185951 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:00:24.186008 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:24.186010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.186606 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.186607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:24.187112 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:24.187114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.187442 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.187443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:24.187506 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:24.187904 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:00:24.187908 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:00:24.189137 (+ 1229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:00:24.190680 (+ 1543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:24.190681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:24.190681 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:24.190839 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:24.191001 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:24.191002 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:24.191002 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:24.191057 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:24.191275 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:00:24.191276 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:00:24.191317 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 09:00:24.191376 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 09:00:24.191464 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 09:00:24.191565 23158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:24.185672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51024 (local address 127.0.98.61:33421)
0619 09:00:24.185717 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:24.185719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:24.185727 (+ 8us) server_negotiation.cc:407] Connection header received
0619 09:00:24.185750 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:24.185751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:24.185761 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:24.185815 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:24.186080 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.186542 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.187184 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.187388 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.187442 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:24.189213 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:24.189214 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:24.189215 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:24.189223 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:24.190610 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:24.190897 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:24.190898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:24.190899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:24.190952 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:24.191105 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:24.191106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:24.191107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:24.191222 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:24.191275 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:24.191405 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 09:00:24.191472 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 09:00:24.191860 23229 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } has no permanent_uuid. Determining permanent_uuid...
I20260619 09:00:24.192513 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:24.185931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40674 (local address 127.0.98.62:40491)
0619 09:00:24.185968 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:24.185970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:24.185977 (+ 7us) server_negotiation.cc:407] Connection header received
0619 09:00:24.186014 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:24.186015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:24.186024 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:24.186073 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:24.186488 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.186942 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.187576 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.187827 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.187889 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:24.190066 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:24.190067 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:24.190067 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:24.190074 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:24.191349 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:24.191667 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:24.191668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:24.191668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:24.191727 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:24.191912 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:24.191913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:24.191914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:24.192053 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:24.192258 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:24.192341 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 09:00:24.192422 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Jun 19 09:00:24 dist-test-slave-h4g5 krb5kdc[22998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 09:00:24 dist-test-slave-h4g5 krb5kdc[22998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 09:00:24.198148 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:24.192136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35569 (local address 127.0.0.1:56236)
0619 09:00:24.192206 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:24.192216 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:00:24.192256 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:00:24.192363 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:00:24.192363 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:00:24.192500 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:00:24.192535 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:24.192536 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.193396 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.193397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:24.193981 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:24.193983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.194325 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.194326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:24.194378 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:24.194753 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:00:24.194756 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:00:24.195922 (+ 1166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:00:24.197291 (+ 1369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:24.197291 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:24.197292 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:24.197434 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:24.197607 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:24.197607 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:24.197608 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:24.197657 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:24.197852 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:00:24.197853 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:00:24.197889 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 09:00:24.197948 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 09:00:24.198042 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 09:00:24.198200 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:24.192176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56236 (local address 127.0.98.60:35569)
0619 09:00:24.192212 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:24.192214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:24.192243 (+ 29us) server_negotiation.cc:407] Connection header received
0619 09:00:24.192299 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:24.192299 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:24.192309 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:24.192351 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:24.192593 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.193333 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.194052 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.194264 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.194315 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:24.195992 (+ 1677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:24.195993 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:24.195993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:24.196000 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:24.197226 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:24.197508 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:24.197509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:24.197509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:24.197557 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:24.197704 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:24.197705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:24.197705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:24.197804 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:24.197853 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:24.197971 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 09:00:24.198089 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":17}
I20260619 09:00:24.199677 23229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2622848d3442475bb10cd797468d22d3: Bootstrap starting.
I20260619 09:00:24.200343 23158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:24.193086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51026 (local address 127.0.98.61:33421)
0619 09:00:24.193123 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:24.193124 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:24.193189 (+ 65us) server_negotiation.cc:407] Connection header received
0619 09:00:24.193246 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:24.193246 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:24.193253 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:24.193292 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:24.193730 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.194285 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.195034 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.195280 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.195329 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:24.197917 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:24.197919 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:24.197920 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:24.197926 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:24.199310 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:24.199643 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:24.199644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:24.199644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:24.199691 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:24.199860 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:24.199861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:24.199861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:24.199960 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:24.200063 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:24.200218 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 09:00:24.200276 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 09:00:24.200687 23229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2622848d3442475bb10cd797468d22d3: Neither blocks nor log segments found. Creating new log.
I20260619 09:00:24.200958 23229 log.cc:824] T 00000000000000000000000000000000 P 2622848d3442475bb10cd797468d22d3: Log is configured to *not* fsync() on all Append() calls
I20260619 09:00:24.201691 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:00:24.201756 23229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2622848d3442475bb10cd797468d22d3: No bootstrap required, opened a new log
I20260619 09:00:24.202478 23053 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9a709cbcfe8b416c9a752dbf3d4005ca"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40674
I20260619 09:00:24.203284 23229 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2622848d3442475bb10cd797468d22d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a709cbcfe8b416c9a752dbf3d4005ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } } peers { permanent_uuid: "f05e92e62bec49ecbfb26442e3f3e8f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33421 } } peers { permanent_uuid: "2622848d3442475bb10cd797468d22d3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } }
I20260619 09:00:24.203403 23229 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2622848d3442475bb10cd797468d22d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2622848d3442475bb10cd797468d22d3, State: Initialized, Role: FOLLOWER
I20260619 09:00:24.203485 23229 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2622848d3442475bb10cd797468d22d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a709cbcfe8b416c9a752dbf3d4005ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } } peers { permanent_uuid: "f05e92e62bec49ecbfb26442e3f3e8f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33421 } } peers { permanent_uuid: "2622848d3442475bb10cd797468d22d3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } }
I20260619 09:00:24.203743 23235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2622848d3442475bb10cd797468d22d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a709cbcfe8b416c9a752dbf3d4005ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } } peers { permanent_uuid: "f05e92e62bec49ecbfb26442e3f3e8f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33421 } } peers { permanent_uuid: "2622848d3442475bb10cd797468d22d3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } } }
I20260619 09:00:24.203797 23229 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2622848d3442475bb10cd797468d22d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:00:24.203840 23235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2622848d3442475bb10cd797468d22d3 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 09:00:24.205516 23246 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2622848d3442475bb10cd797468d22d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 09:00:24.205801 23246 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 09:00:24.205945 23161 master_runner.cc:186] Error getting master registration for 127.0.98.62:40491: OK, instance_id { permanent_uuid: "9a709cbcfe8b416c9a752dbf3d4005ca" instance_seqno: 1781859623874200 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 09:00:24.208202 23161 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 09:00:24.216877 23079 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } attempt: 2
Jun 19 09:00:24 dist-test-slave-h4g5 krb5kdc[22998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 09:00:24.225111 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:24.217444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56252 (local address 127.0.98.60:35569)
0619 09:00:24.217488 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:24.217490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:24.217497 (+ 7us) server_negotiation.cc:407] Connection header received
0619 09:00:24.217521 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:24.217522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:24.217531 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:24.217575 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:24.217932 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.218673 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.219330 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.219528 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.219583 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:24.221647 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:24.221648 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:24.221649 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:24.221659 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:24.223599 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:24.224041 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:24.224043 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:24.224043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:24.224139 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:24.224388 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:24.224389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:24.224390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:24.224687 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:24.224771 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:24.224921 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 09:00:24.225011 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 09:00:24.225109 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:24.217221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35569 (local address 127.0.0.1:56252)
0619 09:00:24.217287 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:24.217300 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 09:00:24.217363 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:00:24.217595 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:00:24.217596 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:00:24.217795 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:00:24.217846 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:24.217848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.218756 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.218757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:24.219252 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:24.219254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.219587 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.219588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:24.219641 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:24.220056 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:00:24.220061 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:00:24.221521 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:00:24.223705 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:24.223706 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:24.223707 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:24.223940 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:24.224207 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:24.224208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:24.224208 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:24.224279 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:24.224762 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:00:24.224763 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:00:24.224816 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 09:00:24.224889 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 09:00:24.224997 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260619 09:00:24.226816 23079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a709cbcfe8b416c9a752dbf3d4005ca: Bootstrap starting.
I20260619 09:00:24.227573 23079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a709cbcfe8b416c9a752dbf3d4005ca: Neither blocks nor log segments found. Creating new log.
I20260619 09:00:24.227877 23079 log.cc:824] T 00000000000000000000000000000000 P 9a709cbcfe8b416c9a752dbf3d4005ca: Log is configured to *not* fsync() on all Append() calls
I20260619 09:00:24.228685 23079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a709cbcfe8b416c9a752dbf3d4005ca: No bootstrap required, opened a new log
I20260619 09:00:24.230032 23079 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a709cbcfe8b416c9a752dbf3d4005ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a709cbcfe8b416c9a752dbf3d4005ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } } peers { permanent_uuid: "f05e92e62bec49ecbfb26442e3f3e8f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33421 } } peers { permanent_uuid: "2622848d3442475bb10cd797468d22d3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } }
I20260619 09:00:24.230181 23079 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a709cbcfe8b416c9a752dbf3d4005ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a709cbcfe8b416c9a752dbf3d4005ca, State: Initialized, Role: FOLLOWER
I20260619 09:00:24.230265 23079 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a709cbcfe8b416c9a752dbf3d4005ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a709cbcfe8b416c9a752dbf3d4005ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } } peers { permanent_uuid: "f05e92e62bec49ecbfb26442e3f3e8f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33421 } } peers { permanent_uuid: "2622848d3442475bb10cd797468d22d3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } }
I20260619 09:00:24.230521 23249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a709cbcfe8b416c9a752dbf3d4005ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a709cbcfe8b416c9a752dbf3d4005ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } } peers { permanent_uuid: "f05e92e62bec49ecbfb26442e3f3e8f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33421 } } peers { permanent_uuid: "2622848d3442475bb10cd797468d22d3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } } }
I20260619 09:00:24.230573 23079 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a709cbcfe8b416c9a752dbf3d4005ca [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:00:24.230602 23249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a709cbcfe8b416c9a752dbf3d4005ca [sys.catalog]: This master's current role is: FOLLOWER
I20260619 09:00:24.231887 23053 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a709cbcfe8b416c9a752dbf3d4005ca [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 09:00:24.232164 23053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a709cbcfe8b416c9a752dbf3d4005ca [term 0 FOLLOWER]: Advancing to term 1
W20260619 09:00:24.232497 23260 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9a709cbcfe8b416c9a752dbf3d4005ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 09:00:24.232553 23260 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 09:00:24.233392 23053 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a709cbcfe8b416c9a752dbf3d4005ca [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a709cbcfe8b416c9a752dbf3d4005ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } } peers { permanent_uuid: "f05e92e62bec49ecbfb26442e3f3e8f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33421 } } peers { permanent_uuid: "2622848d3442475bb10cd797468d22d3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } }
I20260619 09:00:24.233628 23053 leader_election.cc:288] T 00000000000000000000000000000000 P 9a709cbcfe8b416c9a752dbf3d4005ca [CANDIDATE]: Term 1 election: Requested vote from peers f05e92e62bec49ecbfb26442e3f3e8f6 (127.0.98.61:33421), 2622848d3442475bb10cd797468d22d3 (127.0.98.60:35569)
I20260619 09:00:24.234477 23203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9a709cbcfe8b416c9a752dbf3d4005ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2622848d3442475bb10cd797468d22d3"
I20260619 09:00:24.234609 23203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2622848d3442475bb10cd797468d22d3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:00:24.236465 23011 master_runner.cc:186] Error getting master registration for 127.0.98.61:33421: OK, instance_id { permanent_uuid: "f05e92e62bec49ecbfb26442e3f3e8f6" instance_seqno: 1781859624028950 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 09:00:24.238714 23011 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 09:00:24.239240 23203 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2622848d3442475bb10cd797468d22d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a709cbcfe8b416c9a752dbf3d4005ca in term 1.
I20260619 09:00:24.239552 23030 leader_election.cc:302] T 00000000000000000000000000000000 P 9a709cbcfe8b416c9a752dbf3d4005ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2622848d3442475bb10cd797468d22d3, 9a709cbcfe8b416c9a752dbf3d4005ca; no voters:
I20260619 09:00:24.239681 23249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a709cbcfe8b416c9a752dbf3d4005ca [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:00:24.239821 23249 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a709cbcfe8b416c9a752dbf3d4005ca [term 1 LEADER]: Becoming Leader. State: Replica: 9a709cbcfe8b416c9a752dbf3d4005ca, State: Running, Role: LEADER
I20260619 09:00:24.239902 23249 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a709cbcfe8b416c9a752dbf3d4005ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a709cbcfe8b416c9a752dbf3d4005ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } } peers { permanent_uuid: "f05e92e62bec49ecbfb26442e3f3e8f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33421 } } peers { permanent_uuid: "2622848d3442475bb10cd797468d22d3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } }
I20260619 09:00:24.240331 23249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a709cbcfe8b416c9a752dbf3d4005ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a709cbcfe8b416c9a752dbf3d4005ca. Latest consensus state: current_term: 1 leader_uuid: "9a709cbcfe8b416c9a752dbf3d4005ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a709cbcfe8b416c9a752dbf3d4005ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } } peers { permanent_uuid: "f05e92e62bec49ecbfb26442e3f3e8f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33421 } } peers { permanent_uuid: "2622848d3442475bb10cd797468d22d3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } } }
I20260619 09:00:24.240414 23249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a709cbcfe8b416c9a752dbf3d4005ca [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:24.240835 23271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:00:24.241181 23271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:00:24.243458 23203 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2622848d3442475bb10cd797468d22d3 [term 1 FOLLOWER]: Refusing update from remote peer 9a709cbcfe8b416c9a752dbf3d4005ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:00:24.243944 23249 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9a709cbcfe8b416c9a752dbf3d4005ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "2622848d3442475bb10cd797468d22d3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:00:24.246376 23235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2622848d3442475bb10cd797468d22d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a709cbcfe8b416c9a752dbf3d4005ca. Latest consensus state: current_term: 1 leader_uuid: "9a709cbcfe8b416c9a752dbf3d4005ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a709cbcfe8b416c9a752dbf3d4005ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } } peers { permanent_uuid: "f05e92e62bec49ecbfb26442e3f3e8f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33421 } } peers { permanent_uuid: "2622848d3442475bb10cd797468d22d3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } } }
I20260619 09:00:24.246474 23235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2622848d3442475bb10cd797468d22d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 09:00:24.249521 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:24.234699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56260 (local address 127.0.98.60:35569)
0619 09:00:24.234738 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:24.234740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:24.235627 (+ 887us) server_negotiation.cc:407] Connection header received
0619 09:00:24.235936 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:24.235937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:24.235948 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:24.235978 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:24.239243 (+ 3265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.239966 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.243619 (+ 3653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.243921 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.243965 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:24.246726 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:24.246728 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:24.246728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:24.246736 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:24.248328 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:24.248711 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:24.248712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:24.248712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:24.248765 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:24.248942 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:24.248943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:24.248944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:24.249065 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:24.249138 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:24.249345 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 09:00:24.249426 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 09:00:24.253610 23249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a709cbcfe8b416c9a752dbf3d4005ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a709cbcfe8b416c9a752dbf3d4005ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a709cbcfe8b416c9a752dbf3d4005ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } } peers { permanent_uuid: "f05e92e62bec49ecbfb26442e3f3e8f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33421 } } peers { permanent_uuid: "2622848d3442475bb10cd797468d22d3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } } }
I20260619 09:00:24.253721 23249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a709cbcfe8b416c9a752dbf3d4005ca [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:24.253788 23235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2622848d3442475bb10cd797468d22d3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a709cbcfe8b416c9a752dbf3d4005ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a709cbcfe8b416c9a752dbf3d4005ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } } peers { permanent_uuid: "f05e92e62bec49ecbfb26442e3f3e8f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33421 } } peers { permanent_uuid: "2622848d3442475bb10cd797468d22d3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } } }
I20260619 09:00:24.253865 23235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2622848d3442475bb10cd797468d22d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 09:00:24.253961 23273 mvcc.cc:205] Tried to move back new op lower bound from 7298497020899069952 to 7298497020888182784. Current Snapshot: MvccSnapshot[applied={T|T < 7298497020899069952}]
I20260619 09:00:24.254098 23272 mvcc.cc:205] Tried to move back new op lower bound from 7298497020899069952 to 7298497020888182784. Current Snapshot: MvccSnapshot[applied={T|T < 7298497020899069952}]
I20260619 09:00:24.254631 23158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:24.235334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51030 (local address 127.0.98.61:33421)
0619 09:00:24.235374 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:24.235376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:24.237609 (+ 2233us) server_negotiation.cc:407] Connection header received
0619 09:00:24.237698 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:24.237699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:24.237709 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:24.237737 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:24.241925 (+ 4188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.242635 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.246039 (+ 3404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.246373 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.246415 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:24.250206 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:24.250208 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:24.250209 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:24.250217 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:24.251824 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:24.252520 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:24.252521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:24.252522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:24.252800 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:24.253015 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:24.253016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:24.253016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:24.253154 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:24.253248 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:24.254429 (+ 1181us) server_negotiation.cc:299] Negotiation successful
0619 09:00:24.254534 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 09:00:24.255131 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:24.235040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40680 (local address 127.0.98.62:40491)
0619 09:00:24.235076 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:24.235077 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:24.235774 (+ 697us) server_negotiation.cc:407] Connection header received
0619 09:00:24.236831 (+ 1057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:24.236832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:24.236841 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:24.236868 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:24.238415 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.239133 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.244385 (+ 5252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.244690 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.244732 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:24.251786 (+ 7054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:24.251788 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:24.251789 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:24.251798 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:24.253387 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:24.254025 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:24.254026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:24.254026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:24.254083 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:24.254388 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:24.254388 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:24.254389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:24.254534 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:24.254622 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:24.254943 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 09:00:24.255024 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 09:00:24.256492 23271 catalog_manager.cc:1347] Generated new cluster ID: 87fb1412291740f8beea7394dedf86fb
I20260619 09:00:24.256539 23271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:00:24.263010 23154 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } attempt: 3
Jun 19 09:00:24 dist-test-slave-h4g5 krb5kdc[22998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 09:00:24.268836 23271 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:00:24.270036 23271 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:00:24.271412 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:24.263274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35569 (local address 127.0.0.1:56272)
0619 09:00:24.263322 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:24.263333 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:00:24.263370 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:00:24.263500 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:00:24.263501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:00:24.263688 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:00:24.263750 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:24.263752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.264693 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.264695 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:24.265409 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:24.265412 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.265916 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.265917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:24.265994 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:24.266535 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:00:24.266541 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:00:24.268150 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:00:24.270177 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:24.270178 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:24.270179 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:24.270412 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:24.270686 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:24.270687 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:24.270687 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:24.270760 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:24.271092 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:00:24.271093 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:00:24.271156 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 09:00:24.271213 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 09:00:24.271312 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 09:00:24.272989 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:24.263322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56272 (local address 127.0.98.60:35569)
0619 09:00:24.263402 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:24.263404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:24.263410 (+ 6us) server_negotiation.cc:407] Connection header received
0619 09:00:24.263431 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:24.263431 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:24.263440 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:24.263497 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:24.263815 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.264613 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.265475 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:24.265790 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:24.265846 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:24.268260 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:24.268262 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:24.268262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:24.268270 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:24.269986 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:24.270533 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:24.270535 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:24.270535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:24.270609 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:24.270846 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:24.270847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:24.270847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:24.270990 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:24.271043 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:24.272804 (+ 1761us) server_negotiation.cc:299] Negotiation successful
0619 09:00:24.272891 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 09:00:24.274530 23154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f05e92e62bec49ecbfb26442e3f3e8f6: Bootstrap starting.
I20260619 09:00:24.275271 23154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f05e92e62bec49ecbfb26442e3f3e8f6: Neither blocks nor log segments found. Creating new log.
I20260619 09:00:24.275581 23154 log.cc:824] T 00000000000000000000000000000000 P f05e92e62bec49ecbfb26442e3f3e8f6: Log is configured to *not* fsync() on all Append() calls
I20260619 09:00:24.276554 23154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f05e92e62bec49ecbfb26442e3f3e8f6: No bootstrap required, opened a new log
I20260619 09:00:24.278522 23154 raft_consensus.cc:348] T 00000000000000000000000000000000 P f05e92e62bec49ecbfb26442e3f3e8f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a709cbcfe8b416c9a752dbf3d4005ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } } peers { permanent_uuid: "f05e92e62bec49ecbfb26442e3f3e8f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33421 } } peers { permanent_uuid: "2622848d3442475bb10cd797468d22d3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } }
I20260619 09:00:24.278681 23154 raft_consensus.cc:729] T 00000000000000000000000000000000 P f05e92e62bec49ecbfb26442e3f3e8f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f05e92e62bec49ecbfb26442e3f3e8f6, State: Initialized, Role: FOLLOWER
I20260619 09:00:24.278764 23154 consensus_queue.cc:260] T 00000000000000000000000000000000 P f05e92e62bec49ecbfb26442e3f3e8f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a709cbcfe8b416c9a752dbf3d4005ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } } peers { permanent_uuid: "f05e92e62bec49ecbfb26442e3f3e8f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33421 } } peers { permanent_uuid: "2622848d3442475bb10cd797468d22d3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } }
I20260619 09:00:24.279155 23127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f05e92e62bec49ecbfb26442e3f3e8f6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:00:24.279525 23271 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a709cbcfe8b416c9a752dbf3d4005ca: Generated new TSK 0
I20260619 09:00:24.279770 23271 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:00:24.280067 23127 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f05e92e62bec49ecbfb26442e3f3e8f6 [term 1 FOLLOWER]: Refusing update from remote peer 9a709cbcfe8b416c9a752dbf3d4005ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:00:24.279175 23128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9a709cbcfe8b416c9a752dbf3d4005ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f05e92e62bec49ecbfb26442e3f3e8f6"
I20260619 09:00:24.280481 23269 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9a709cbcfe8b416c9a752dbf3d4005ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "f05e92e62bec49ecbfb26442e3f3e8f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:00:24.280985 23128 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f05e92e62bec49ecbfb26442e3f3e8f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a709cbcfe8b416c9a752dbf3d4005ca in term 1.
I20260619 09:00:24.279106 23154 sys_catalog.cc:564] T 00000000000000000000000000000000 P f05e92e62bec49ecbfb26442e3f3e8f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:00:24.279073 23280 sys_catalog.cc:455] T 00000000000000000000000000000000 P f05e92e62bec49ecbfb26442e3f3e8f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a709cbcfe8b416c9a752dbf3d4005ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } } peers { permanent_uuid: "f05e92e62bec49ecbfb26442e3f3e8f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33421 } } peers { permanent_uuid: "2622848d3442475bb10cd797468d22d3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } } }
I20260619 09:00:24.283658 23280 sys_catalog.cc:458] T 00000000000000000000000000000000 P f05e92e62bec49ecbfb26442e3f3e8f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 09:00:24.283752 23280 sys_catalog.cc:455] T 00000000000000000000000000000000 P f05e92e62bec49ecbfb26442e3f3e8f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a709cbcfe8b416c9a752dbf3d4005ca. Latest consensus state: current_term: 1 leader_uuid: "9a709cbcfe8b416c9a752dbf3d4005ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a709cbcfe8b416c9a752dbf3d4005ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } } peers { permanent_uuid: "f05e92e62bec49ecbfb26442e3f3e8f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33421 } } peers { permanent_uuid: "2622848d3442475bb10cd797468d22d3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } } }
I20260619 09:00:24.283854 23280 sys_catalog.cc:458] T 00000000000000000000000000000000 P f05e92e62bec49ecbfb26442e3f3e8f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 09:00:24.283913 23280 sys_catalog.cc:455] T 00000000000000000000000000000000 P f05e92e62bec49ecbfb26442e3f3e8f6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a709cbcfe8b416c9a752dbf3d4005ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a709cbcfe8b416c9a752dbf3d4005ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } } peers { permanent_uuid: "f05e92e62bec49ecbfb26442e3f3e8f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33421 } } peers { permanent_uuid: "2622848d3442475bb10cd797468d22d3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } } }
I20260619 09:00:24.283993 23280 sys_catalog.cc:458] T 00000000000000000000000000000000 P f05e92e62bec49ecbfb26442e3f3e8f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 09:00:24.288760 23283 mvcc.cc:205] Tried to move back new op lower bound from 7298497021043494912 to 7298497020888182784. Current Snapshot: MvccSnapshot[applied={T|T < 7298497020899069952}]
W20260619 09:00:24.289283 23295 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f05e92e62bec49ecbfb26442e3f3e8f6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 09:00:24.289341 23295 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260619 09:00:24.304634 23082 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:00:24.302530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40694 (local address 127.0.98.62:40491)
0619 09:00:24.302576 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:24.302578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:24.303610 (+ 1032us) server_negotiation.cc:407] Connection header received
0619 09:00:24.303724 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:24.303725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:24.303737 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:24.303767 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:24.304554 (+ 787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40694: BlockingRecv error: recv got EOF from 127.0.0.1:40694 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260619 09:00:24.305161 23232 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:00:24.302760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56286 (local address 127.0.98.60:35569)
0619 09:00:24.302801 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:24.302803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:24.304219 (+ 1416us) server_negotiation.cc:407] Connection header received
0619 09:00:24.304313 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:24.304314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:24.304325 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:24.304352 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:24.305092 (+ 740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56286: BlockingRecv error: recv got EOF from 127.0.0.1:56286 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260619 09:00:24.305878 23158 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:00:24.303121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51036 (local address 127.0.98.61:33421)
0619 09:00:24.303164 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:24.303165 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:24.305383 (+ 2218us) server_negotiation.cc:407] Connection header received
0619 09:00:24.305477 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:24.305478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:24.305490 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:24.305518 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:24.305818 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51036: BlockingRecv error: recv got EOF from 127.0.0.1:51036 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260619 09:00:24.308382 23158 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:00:24.307533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51042 (local address 127.0.98.61:33421)
0619 09:00:24.307568 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:24.307570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:24.307784 (+ 214us) server_negotiation.cc:407] Connection header received
0619 09:00:24.307891 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:24.307892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:24.307910 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:24.307939 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:24.308331 (+ 392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51042: BlockingRecv error: recv got EOF from 127.0.0.1:51042 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260619 09:00:24.311514 23158 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:00:24.310717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51046 (local address 127.0.98.61:33421)
0619 09:00:24.310749 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:24.310751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:24.310999 (+ 248us) server_negotiation.cc:407] Connection header received
0619 09:00:24.311105 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:24.311106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:24.311118 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:24.311146 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:24.311464 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51046: BlockingRecv error: recv got EOF from 127.0.0.1:51046 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260619 09:00:24.315634 23158 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:00:24.314781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51058 (local address 127.0.98.61:33421)
0619 09:00:24.314815 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:24.314816 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:24.315126 (+ 310us) server_negotiation.cc:407] Connection header received
0619 09:00:24.315227 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:24.315228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:24.315239 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:24.315267 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:24.315584 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51058: BlockingRecv error: recv got EOF from 127.0.0.1:51058 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260619 09:00:24.321725 23158 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:00:24.320899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51068 (local address 127.0.98.61:33421)
0619 09:00:24.320936 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:24.320938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:24.321194 (+ 256us) server_negotiation.cc:407] Connection header received
0619 09:00:24.321290 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:24.321291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:24.321302 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:24.321330 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:24.321675 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51068: BlockingRecv error: recv got EOF from 127.0.0.1:51068 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260619 09:00:24.332051 23158 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:00:24.331052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51072 (local address 127.0.98.61:33421)
0619 09:00:24.331150 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:24.331153 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:24.331475 (+ 322us) server_negotiation.cc:407] Connection header received
0619 09:00:24.331577 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:24.331578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:24.331590 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:24.331618 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:24.331989 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51072: BlockingRecv error: recv got EOF from 127.0.0.1:51072 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 09:00:24.350498 23158 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:00:24.349460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51080 (local address 127.0.98.61:33421)
0619 09:00:24.349506 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:24.349508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:24.349828 (+ 320us) server_negotiation.cc:407] Connection header received
0619 09:00:24.349937 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:24.349938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:24.349951 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:24.349981 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:24.350414 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51080: BlockingRecv error: recv got EOF from 127.0.0.1:51080 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 09:00:24.384428 23158 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:00:24.383717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51090 (local address 127.0.98.61:33421)
0619 09:00:24.383763 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:24.383765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:24.383879 (+ 114us) server_negotiation.cc:407] Connection header received
0619 09:00:24.383935 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:24.383935 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:24.383947 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:24.383995 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:24.384366 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51090: BlockingRecv error: recv got EOF from 127.0.0.1:51090 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260619 09:00:24.450281 23158 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:00:24.449527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51098 (local address 127.0.98.61:33421)
0619 09:00:24.449592 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:24.449593 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:24.449657 (+ 64us) server_negotiation.cc:407] Connection header received
0619 09:00:24.449742 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:24.449743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:24.449754 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:24.449793 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:24.450216 (+ 423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51098: BlockingRecv error: recv got EOF from 127.0.0.1:51098 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 09:00:24.579995 23158 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:00:24.579345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51112 (local address 127.0.98.61:33421)
0619 09:00:24.579418 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:24.579420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:24.579481 (+ 61us) server_negotiation.cc:407] Connection header received
0619 09:00:24.579552 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:24.579553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:24.579565 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:24.579614 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:24.579936 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51112: BlockingRecv error: recv got EOF from 127.0.0.1:51112 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 09:00:24.832862 23354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a709cbcfe8b416c9a752dbf3d4005ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a709cbcfe8b416c9a752dbf3d4005ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a709cbcfe8b416c9a752dbf3d4005ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40491 } } peers { permanent_uuid: "f05e92e62bec49ecbfb26442e3f3e8f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33421 } } peers { permanent_uuid: "2622848d3442475bb10cd797468d22d3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35569 } } }
I20260619 09:00:24.833006 23354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a709cbcfe8b416c9a752dbf3d4005ca [sys.catalog]: This master's current role is: LEADER
W20260619 09:00:24.837776 23158 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:00:24.837182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51122 (local address 127.0.98.61:33421)
0619 09:00:24.837244 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:24.837246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:24.837259 (+ 13us) server_negotiation.cc:407] Connection header received
0619 09:00:24.837339 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:24.837340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:24.837351 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:24.837398 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:24.837714 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51122: BlockingRecv error: recv got EOF from 127.0.0.1:51122 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 09:00:25.206312 23246 catalog_manager.cc:1259] Loaded cluster ID: 87fb1412291740f8beea7394dedf86fb
I20260619 09:00:25.206384 23246 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2622848d3442475bb10cd797468d22d3: loading cluster ID for follower catalog manager: success
I20260619 09:00:25.207227 23246 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2622848d3442475bb10cd797468d22d3: acquiring CA information for follower catalog manager: success
I20260619 09:00:25.207552 23246 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2622848d3442475bb10cd797468d22d3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 09:00:25.289709 23295 catalog_manager.cc:1259] Loaded cluster ID: 87fb1412291740f8beea7394dedf86fb
I20260619 09:00:25.289793 23295 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f05e92e62bec49ecbfb26442e3f3e8f6: loading cluster ID for follower catalog manager: success
I20260619 09:00:25.290637 23295 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f05e92e62bec49ecbfb26442e3f3e8f6: acquiring CA information for follower catalog manager: success
I20260619 09:00:25.290937 23295 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f05e92e62bec49ecbfb26442e3f3e8f6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 09:00:25.353210 23367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:25.350865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51130 (local address 127.0.98.61:33421)
0619 09:00:25.350991 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:25.350993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:25.351004 (+ 11us) server_negotiation.cc:407] Connection header received
0619 09:00:25.351031 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:25.351032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:25.351081 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:25.351152 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:00:25.351275 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:25.351776 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:25.352577 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:25.352772 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:25.352825 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:25.352936 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:00:25.353043 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:00:25.353096 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:25.353113 (+ 17us) server_negotiation.cc:299] Negotiation successful
0619 09:00:25.353129 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":67,"threads_started":1}
I20260619 09:00:25.357345 23373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:25.354850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56290 (local address 127.0.98.60:35569)
0619 09:00:25.354949 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:25.354951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:25.354963 (+ 12us) server_negotiation.cc:407] Connection header received
0619 09:00:25.355027 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:25.355027 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:25.355060 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:25.355124 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:00:25.355188 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:25.355688 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:25.356591 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:25.356782 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:25.356851 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:25.357069 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:00:25.357173 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:00:25.357250 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:25.357265 (+ 15us) server_negotiation.cc:299] Negotiation successful
0619 09:00:25.357283 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":52,"threads_started":1}
I20260619 09:00:25.361559 23380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:25.358948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40696 (local address 127.0.98.62:40491)
0619 09:00:25.359039 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:25.359041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:25.359048 (+ 7us) server_negotiation.cc:407] Connection header received
0619 09:00:25.359093 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:25.359094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:25.359146 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:25.359246 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:00:25.359484 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:25.360007 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:25.360774 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:25.360975 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:25.361029 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:25.361230 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:00:25.361339 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:00:25.361403 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:25.361453 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 09:00:25.361477 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":44,"threads_started":1}
I20260619 09:00:25.361737 23373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:25.358896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56302 (local address 127.0.98.60:35569)
0619 09:00:25.358943 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:25.358945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:25.358953 (+ 8us) server_negotiation.cc:407] Connection header received
0619 09:00:25.359073 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:25.359074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:25.359084 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:25.359142 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:00:25.359264 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:25.359921 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:25.361018 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:25.361260 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:25.361313 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:25.361441 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:00:25.361471 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:00:25.361511 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:25.361644 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 09:00:25.361667 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 09:00:25.361860 23367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:25.358973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51144 (local address 127.0.98.61:33421)
0619 09:00:25.359010 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:25.359012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:25.359220 (+ 208us) server_negotiation.cc:407] Connection header received
0619 09:00:25.359251 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:25.359252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:25.359262 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:25.359312 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:00:25.359454 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:25.360156 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:25.361122 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:25.361384 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:25.361437 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:25.361620 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:00:25.361650 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:00:25.361676 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:25.361772 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 09:00:25.361797 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 09:00:25.363022 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 23011
I20260619 09:00:25.367532 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 23086
I20260619 09:00:25.371483 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5ocqKJ/build/release/bin/kudu with pid 23161
2026-06-19T09:00:25Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2683 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5412 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (263835 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST